Extracting Fine Grain Labels from Medical Imaging Reports

ABSTRACT

Mechanisms are provided to implement a fine-grained finding descriptor generation computing tool that automatically generates fine-grained labels for downstream computer system operations. The mechanisms process medical report content based on a core finding lexicon, to extract core finding instances from the medical report content. The mechanisms execute, for each core finding instance, automated computer NLP operations that generate a parse tree for the portion of the medical report content corresponding to the core finding instance, perform phrasal grouping on the parse tree to thereby associate one or more modifiers of core findings specified in the portion of the medical report content with the core finding instance, and generate a fine-grained finding descriptor data structure for the core finding instance based on the association of one or more modifiers of the core finding with the core finding instance.

BACKGROUND

The present application relates generally to an improved data processingapparatus and method, and more specifically to mechanisms for extractingfine grain labels from medical imaging reports.

Leveraging machine learning capabilities of modern computing devices toassist with pattern recognition in medical image analysis is a focus ofgreat attention in modern medical innovations. However, the quality oflearning that is able to be performed by such machine learning is afunction of the granularity of labels that can be attached to themedical images. Currently, only coarse-grained finding labels are ableto be used with any success, making such approaches of significantlylimited use in clinical practice.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described herein in the DetailedDescription. This Summary is not intended to identify key factors oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

In one illustrative embodiment, a method, in a data processing systemspecifically configured to implement a fine-grained finding descriptorgeneration computing tool that automatically generates fine-grainedlabels for downstream computer system operations. The method comprisesprocessing, by the fine-grained finding descriptor generation computingtool, medical report natural language content of at least one medicalimaging report data structure associated with at least one medicalimage, based on a core finding lexicon data structure, to extract a setof core finding instances of one or more core findings in the corefinding lexicon data structure, from the medical report natural languagecontent. The one or more core findings are terms describing one ofanatomical structures or abnormalities present in the at least onemedical image. The method also comprises executing, by the fine-grainedfinding descriptor generation computing tool, for each core findinginstance in the extracted set of core finding instances, automatedcomputer natural language processing operations comprising: (1)generating a parse tree data structure for a corresponding portion ofthe medical report natural language content corresponding to the corefinding instance; (2) automatically executing phrasal grouping computeroperations on the parse tree data structure to thereby associate one ormore modifiers of core findings specified in the portion of the medicalreport natural language content with the core finding instance, whereinthe one or more modifiers are terms further defining a characteristic ofthe core finding; and (3) generating, by the fine-grained findingdescriptor generation computing tool, a fine-grained finding descriptordata structure for the core finding instance based on the association ofone or more modifiers of the core finding with the core findinginstance. The method also comprises storing the fine-grained findingdescriptor data structure in a fine-grained finding descriptor databasefor use in downstream computer system operations.

In other illustrative embodiments, a computer program product comprisinga computer useable or readable medium having a computer readable programis provided. The computer readable program, when executed on a computingdevice, causes the computing device to perform various ones of, andcombinations of, the operations outlined above with regard to the methodillustrative embodiment.

In yet another illustrative embodiment, a system/apparatus is provided.The system/apparatus may comprise one or more processors and a memorycoupled to the one or more processors. The memory may compriseinstructions which, when executed by the one or more processors, causethe one or more processors to perform various ones of, and combinationsof, the operations outlined above with regard to the method illustrativeembodiment.

These and other features and advantages of the present invention will bedescribed in, or will become apparent to those of ordinary skill in theart in view of, the following detailed description of the exampleembodiments of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, as well as a preferred mode of use and further objectivesand advantages thereof, will best be understood by reference to thefollowing detailed description of illustrative embodiments when read inconjunction with the accompanying drawings, wherein:

FIGS. 1A and 1B are example diagrams of chest X-rays showingcardiomegaly in a patient, with FIG. 1B being a severe case;

FIG. 1C is an example medical imaging report;

FIG. 2 is a diagram showing differences in modifiers associated withdifferent types of core findings in medical reports;

FIG. 3 is a diagram showing concept categories of UMLS relevant forfinding vocabulary generation in accordance with one illustrativeembodiment;

FIG. 4A is a diagram illustrating example core finding labels found by acore findings lexicon development computing tool to be sufficient fordescribing findings in anteroposterior (AP) chest radiographs inaccordance with one illustrative embodiment;

FIG. 4B is another diagram illustrating a portion of a core findingslexicon in which the various columns of information for the corefindings are shown in accordance with one illustrative embodiment;

FIG. 5 is an example diagram illustrating prefix extraction for termswithin a vocabulary phrase to increase specificity of matching inaccordance with one illustrative embodiment;

FIG. 6A illustrates an example of a deterministic algorithm thatidentifies a smallest distinguishable prefix per term in a phrase inaccordance with one illustrative embodiment;

FIG. 6B illustrates an example of a longest common subfix (LCF)algorithm in accordance with one illustrative embodiment;

FIG. 7A illustrates a sample Slot Grammar (SG) parse tree for thesentence “The lungs are normally inflated without evidence of focalairspace disease pleural effusion or pneumothorax” in accordance withone illustrative embodiment;

FIG. 7B illustrates a depiction of a phrasal grouping process using aconnected component analysis in accordance with an illustrativeembodiment;

FIG. 8 illustrates an example of negation detection for the sentence“There is no evidence suggesting that he has cancer” in accordance withone illustrative embodiment;

FIG. 9 provides a listing of examples of types of fine-grained findingdescriptors or labels extracted from sentences from redacted medicalimaging reports in accordance with one illustrative embodiment;

FIG. 10 is an example of a machine learning/deep learning (ML/DL)computer model that may be trained for medical image augmentation(labeling) in accordance with one illustrative embodiment;

FIG. 11 depicts a pictorial representation of an example distributeddata processing system in which aspects of the illustrative embodimentsmay be implemented;

FIG. 12 is a block diagram of one example data processing system inwhich aspects of the illustrative embodiments may be implemented;

FIG. 13 is a flowchart outlining an example operation for generatingfine-grained finding descriptor data structures from medical imagingreports and using those fine-grained finding descriptor data structuresto train a machine learning computer model in accordance with oneillustrative embodiment;

FIG. 14A is an example diagram illustrating an overall automated medicalimaging report generation workflow in accordance with one illustrativeembodiment;

FIG. 14B is another example diagram illustrating an overall automatedmedical imaging report generation workflow with additional detailsregarding report database preparation in accordance with oneillustrative embodiment;

FIG. 15 is an example diagram illustrating examples of medical imagingreports generated using manual processes versus the automated mechanismsof one illustrative embodiment; and

FIG. 16 is flowchart outlining an example operation for automatedmedical imaging report generation in accordance with one illustrativeembodiment.

DETAILED DESCRIPTION

Medical imaging, such ultrasound imaging, magnetic resonance imaging,radiography, computed tomography (CT), etc., is an important part ofmodern medical practices, giving insights into the internal structuresand medical conditions of patients that cannot be otherwise identifiedfrom outside the patient's body. However, medical imaging typicallyrequires highly trained human beings to be able to read captured imagesand apply their own knowledge to what the human being sees in the imagesto make medical findings. This is a significant source of potentialerror, especially when one considers that such highly trained humanbeings, e.g., radiologists or the like, are increasingly being asked toread and report on larger numbers of medical imaging studies inincreasingly shorter amounts of time.

To assist with these medical imaging tasks, computing tools have beendeveloped to perform image analysis and identify coarse grained labelsfor medical images, such as labels identifying opacities, masses, andnodules. However, these coarse-grained labels are insufficientlydescribed to be of use in automated medical imaging reporting. Forexample, using a coarse grained label of “cardiomegaly” as the label forboth the images in FIGS. 1A and 1B is not sufficient to describe theseimages as one constitutes a severe case (FIG. 1B) and may need moreprompt attention and the coarse grained label does not identify anydifferentiation between such cases. Before such computing tools can beincorporated into clinical practices to produce automated preliminaryreads of medical imaging studies, the computing system models need to beable to recognize not only a comprehensive and broad spectrum of medicalimaging findings, but also describe them in a fine grained fashion, suchas covering laterality, anatomical location, severity, appearancecharacteristics, etc. such that distinctions between different types ofthe same coarse grain finding can be made apparent to the medicalpractitioner.

That is, a human generated full-fledged preliminary read radiologyreport, for example, describes various types of findings along withtheir positioning, laterality, severity, appearance characteristics,etc., as determined by a human being manually viewing the medical image.FIG. 1C is an example of one type of preliminary read radiology reportgenerated manually. Currently medical image analysis computing tools areunable to provide such full-fledged preliminary reads of medical imagingstudies and provide automated report generation at a same level ofspecificity as human generated reporting. Thus, while current medicalimage analysis computing tools provide some assistance to the medicalpractitioner, they do not have the level of detailed reporting thatcurrently can only be achieved manually.

Thus, to capture realistic read scenarios, deep learning computermodels, i.e., neural network computer models that learn through amachine learning process implemented on large sets of data, should betrained on fine-grained finding labels, where a “fine grained label” isdistinguished from the “coarse grained labels” in that the fine-grainedfinding labels are able to differentiate different types or sub-types offindings associated with coarse grained labels by providing additionalfinding characteristics, such as type, positive/negative finding, andvarious modifiers. For example, as will be discussed hereafter, in thecontext of the present invention, a fine-grained label, or FFL, may bedenoted by the structure F_(i)=<T_(i)|N_(i)|C_(i)|M_(i)*> where F_(i) isthe FFL, T_(i) is the finding type, N_(i)=yes|no indicates a positive ornegative finding (i.e. is present versus absent), C_(i) is the corefinding itself, and M_(i) are one or more of the possible findingmodifiers. A coarse finding label, or CFL may include only the corefinding itself without the associated attributes of finding type,positive/negative finding, and modifiers.

A number of recent approaches have attempted to take advantage of theassociated medical imaging reports to automatically label thecorresponding images. However, they have been limited to a small numberof coarse grained core findings. Complete labeling of images for allpossible findings, i.e. coarse grained core findings and more finegrained findings differentiating different types of the coarse grainedcore findings, seen in a specific modality of medical imaging is achallenging problem requiring the development of both vocabulariescovering these findings and development of high precision and recallmethods for extracting labels from the medical imaging study'sassociated medical imaging reports which can then be used to label themedical images for review by medical practitioners.

The illustrative embodiments provide an improved automated computer tooland computer tool methodology to automatically extract, throughautomated computer processes without requiring human intervention,fine-grained finding labels from medical imaging reports. The improvedautomated computer tool and computer tool methodology provides acomprehensive approach to extracting the fine-grained finding labelsfrom medical imaging reports, e.g., radiology reports, which implementsa new descriptor for fine-grained finding labels utilizing validcombinations of findings and their characterization modifiers, i.e.terms that characterize attributes of the findings, e.g., positioning,laterality, severity, appearance characteristics, etc., found in medicalimaging reports. The illustrative embodiments further provide avocabulary-driven concept algorithm for automatically finding thesefindings and modifiers from natural language content, e.g., sentences,in the medical imaging reports. The vocabulary for these findings andmodifiers may be derived from established knowledge sources, such asUnified Medical Language System (UMLS) knowledge graphs, or derived fromclinician curated custom lexicons. A phrasal grouping computing toolassociates detailed characterization modifiers with the relevantfindings in the natural language content. Positive and negativeinstances of a finding are separated and overall fine-grained findinglabels are generated from the medical imaging report. These fine-grainedfinding labels may then be utilized to train a deep learning computermodel, such as for labeling medical images, for example, andautomatically generating preliminary read reports for medical imagingstudies.

Although the primary illustrative embodiment described herein will bedescribed with regard to generating fine-grained finding labels fortraining deep learning computer models to perform fine-grained findinglabeling of medical images such that fine-grained findings may beautomatically determined and reported from medical image processing bythe trained deep learning computer model, the illustrative embodimentsare not limited to such. To the contrary, the improved automatedcomputing tool and computing tool methodology of the illustrativeembodiments are applicable to other uses where it is important to have arefined understanding of the semantic context in a textual report, suchas patient medical condition summary generation, for example. Moreover,being able to extract fine-grained finding label information fromclinical reports, and medical imaging reports in particular, can havesignificant implications for clinical care, such as interpretingaffected anatomy from the extracted fine grained finding labelinformation which can trigger the scheduling of an imaging studyrelating to the anatomy in a downstream clinical workflow alert, usingthe extracted fine grained finding label identification to automaticallyset up reminders for appointments and trigger additional billingprocedures based on the severity of the condition, etc.

Moreover, as chest radiographs, such as those shown in FIGS. 1A and 1B,are the most common diagnostic exam in emergency rooms and intensivecare units today, these chest radiographs will be the example basis forexplaining the improvements provided by the automated computer toolmechanisms of the illustrative embodiments. However, it should beappreciated that these are only provided as examples and the presentinvention may be implemented with any type of medical imaging technologycurrently known or later developed, in which textual reports accompanythe medical images. For example, the mechanisms of the illustrativeembodiments may be implemented with medical imaging studies of varioustechnologies including, but not limited to, radiograph (e.g., X-rayradiography), computed tomography (CT), fluoroscopy, magnetic resonanceimaging (MRI), medical ultrasonography or ultrasound, endoscopy,elastography, tactile imaging, thermography, medical photography andnuclear medicine functional imaging techniques, e.g., positron emissiontomography (PET), and the like.

Before beginning the discussion of the various aspects of theillustrative embodiments and the improved computer operations performedby the illustrative embodiments, it should first be appreciated thatthroughout this description the term “mechanism” will be used to referto elements of the present invention that perform various operations,functions, and the like. A “mechanism,” as the term is used herein, maybe an implementation of the functions or aspects of the illustrativeembodiments in the form of an apparatus, a procedure, or a computerprogram product. In the case of a procedure, the procedure isimplemented by one or more devices, apparatus, computers, dataprocessing systems, or the like. In the case of a computer programproduct, the logic represented by computer code or instructions embodiedin or on the computer program product is executed by one or morehardware devices in order to implement the functionality or perform theoperations associated with the specific “mechanism.” Thus, themechanisms described herein may be implemented as specialized hardware,software executing on hardware to thereby configure the hardware toimplement the specialized functionality of the present invention whichthe hardware would not otherwise be able to perform, softwareinstructions stored on a medium such that the instructions are readilyexecutable by hardware to thereby specifically configure the hardware toperform the recited functionality and specific computer operationsdescribed herein, a procedure or method for executing the functions, ora combination of any of the above.

The present description may make reference to “computing tools” or“tools” with corresponding functional descriptors of the computingtools, e.g., core finding lexicon development computing tool. When suchterminology is used herein, the terminology is intended to refer to aspecifically configured computing tool, configured with specificcomputing logic provided in executed software and/or hardware, torealize the function of the functional descriptor. That is, a “corefinding lexicon development computing tool”, for example, is aspecifically configured computing tool that is specifically configuredwith software and/or hardware computing logic that specifically performsthe operations described herein to develop a core finding lexicon. Thesecomputing tools are specialized computing tools that which arespecifically configured to perform the operations to realize thecorresponding function. Thus, these computing tools are not genericcomputing tools performing generic computer operations, but rather arespecialized computing tools performing specialized functions.

The present description and claims may make use of the terms “a”, “atleast one of”, and “one or more of” with regard to particular featuresand elements of the illustrative embodiments. It should be appreciatedthat these terms and phrases are intended to state that there is atleast one of the particular feature or element present in the particularillustrative embodiment, but that more than one can also be present.That is, these terms/phrases are not intended to limit the descriptionor claims to a single feature/element being present or require that aplurality of such features/elements be present. To the contrary, theseterms/phrases only require at least a single feature/element with thepossibility of a plurality of such features/elements being within thescope of the description and claims.

Moreover, it should be appreciated that the use of the term “engine,” ifused herein with regard to describing embodiments and features of theinvention, is not intended to be limiting of any particularimplementation for accomplishing and/or performing the actions, steps,processes, etc., attributable to and/or performed by the engine. Anengine may be, but is not limited to, software, hardware and/or firmwareor any combination thereof that performs the specified functionsincluding, but not limited to, any use of a general and/or specializedprocessor in combination with appropriate software loaded or stored in amachine readable memory and executed by the processor. Further, any nameassociated with a particular engine is, unless otherwise specified, forpurposes of convenience of reference and not intended to be limiting toa specific implementation. Additionally, any functionality attributed toan engine may be equally performed by multiple engines, incorporatedinto and/or combined with the functionality of another engine of thesame or different type, or distributed across one or more engines ofvarious configurations.

In addition, it should be appreciated that the following descriptionuses a plurality of various examples for various elements of theillustrative embodiments to further illustrate example implementationsof the illustrative embodiments and to aid in the understanding of themechanisms of the illustrative embodiments. These examples intended tobe non-limiting and are not exhaustive of the various possibilities forimplementing the mechanisms of the illustrative embodiments. It will beapparent to those of ordinary skill in the art in view of the presentdescription that there are many other alternative implementations forthese various elements that may be utilized in addition to, or inreplacement of, the examples provided herein without departing from thespirit and scope of the present invention.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

As noted above, the illustrative embodiments provide a new improvedautomated computing tool and computing tool methodology that extractsfine-grained finding labels (FFLs) for medical images from medicalimaging reports to thereby automatically learn FFLs that occur inmedical imaging reports such that they can be used to train machinelearning or deep learning (ML/DL) computer models that providespecialized computing tools for performing cognitive (artificialintelligence) computing operations, such as medical image labeling,automated preliminary medical imaging report generation, automatedpatient summary generation, or the like. The automated computing toolmethodology will first be described followed by a description of thecomputing tool architecture. In addition, specific example embodimentsof trained ML/DL models that distinguish FFLs for automated medicalimaging applications and automated preliminary medical image reportgeneration will be described.

Fine-Grained Finding Descriptor and Core Finding Vocabulary

The mechanisms of the illustrative embodiments utilize a new finegrained finding descriptor data structure to represent findings in afine-grained manner with not only the core finding identified, but alsoany finding modifiers and other attributes of the finding, such as typeand positivity attributes. For example, in some illustrativeembodiments, the fine-grained finding descriptor data structure isdefined as F_(i)=<T_(i)|N_(i)|C_(i)|M_(i)*> where F_(i) is thefine-grained label, T_(i) is the finding type, N_(i)=yes|no andindicates a positive or ruled-out finding, C_(i) is the core findingitself, and M_(i) are one or more of the possible finding modifiers. Inthis pattern, each modified M_(i) is at its designated positionseparated by a |. The finding types in chest X-rays, for example, areadequately covered by six major categories namely, anatomical findings,tubes and lines and their placements, external devices,viewpoint-related issues, and implied diseases associated with findings.By analyzing a large set of chest radiology reports, the set of relevantmodifiers M_(i) for each finding type T_(i) may be determined, and arein fact different for each finding type T_(i), as shown in FIG. 2.

In some illustrative embodiments, in order to find a list of validvalues for the core findings C_(i) and modifiers M_(i) for each findingtype T_(i), a semi-automated process may be implemented by a corefindings lexicon development tool to perform both a top-down andbottom-up analysis of medical imaging reports and medical imagingterminology used by medical professionals, to arrive at a vocabulary orlexicon for a particular type of medical imaging, e.g., chest radiologyimages. The clinician-guided processes implement automated computerizednatural language processing computer tools and techniques to analyze andextract features from natural language content, to perform comparisonsand analysis that facilitate identifying terms or phrases, representinglabels of medical image features, that are frequently used to representmedical concepts in medical image reports.

With regard to the top-down analysis, mechanisms are provided to groupkey visual observation labels, e.g., natural language terms or phrases,that medical imaging professionals use in medical imaging reports, intolexically and semantically meaningful groups. These groupings are thencompared to a corpus of best practices literature in order to identifyterms/phrases that represent core findings in each of the finding typecategories. With regard to the bottom-up analysis, one or more corporaof medical imaging reports, such as may be obtained from various sourcesincluding Indiana data hub dataset, a labeled collection created fromNational Institutes of Health (NIH) supplied data, and the MIMIC-4reports, are mined to extract frequently occurring n-grams, i.e. n-gramsoccurring more than a predetermined threshold number of times in themedical reports, that also had a mapping to categories relating the UMLSconcept categories, such as those shown in FIG. 3. The frequentlyoccurring n-grams are then queried against a clinical knowledge basedproviding a large dataset of medical concepts, thereby providing a setof core terms useful or findings vocabulary generation. A core termexpansion tool is used to identify various forms of describing a finding(e.g., infiltrate, infiltration) or alternative ways of describing thesame finding (e.g., “cardiomegaly”, “heart is enlarged”, “enlargedcardiac silhouette”), abbreviations, misspellings, and semanticalequivalent ways of describing the same medical imaging concepts(synonyms and alternate forms), as well as ontologically relatedconcepts.

In one illustrative embodiment, the process used to derive a list ofvalid values for core finding labels and modifiers for each finding typeis a semi-automatic process that involves a clinician-directed curationprocess. Specifically, a team of clinicians (e.g., 3 radiologists and 1internal medicine doctor) used a combination of top-down and bottom-upprocesses to uncover the list of findings seen in anteroposterior (AP)chest radiographs and recorded them in a chest X-ray lexicon. Theclinicians systematically mapped the key visual observations (labels)that radiologists describe in the reports and grouped the labels intolexically and semantically meaningful groups based on their visualappearance similarities. Using a top-down approach, the cliniciansiteratively searched through the best practices literature, includingFleishner Society guidelines, consulted several practicing radiologists,and provided a raw list of everyday use terms from their own practicesto arrive at a list of core findings in each of the finding typecategories. Next, using a bottom-up approach, report collections,derived from a variety of data sources including the Indiana dataset(3000 reports), internally labeled collection created from NIH supplieddata (16,000 reports), and the MIMIC-4 reports (over 180,000 reports).Frequently occurring n-grams, where n varied from 1 to 13, wereextracted that also had a mapping to meaningful categories related tothe UMLS concept categories in FIG. 3. The resulting frequentlyoccurring n-grams were queried against a clinical knowledge databasehaving concepts assembled from reference vocabularies from UMLS, e.g.,70 reference vocabularies. The clinical knowledge database in oneillustrative embodiment had over 5.3 million concepts. This gave rise toa set of core terms useful for findings vocabulary generation, e.g.,1500 core terms in the set of core terms. The core term expansion toolthen expanded this set of core terms by capturing and relating thevarious forms of describing findings, alternative ways of saying thesame finding, abbreviations, misspellings, synonyms, alternate forms,etc., and ontologically related concepts. Each expansion was reviewed bytwo radiologists for agreement resulting in a lexicon consisting ofunique terms covering a space of multiple core findings and multiplemodifier types, where each modifier type may have many differentinstances, e.g., the severity modifier may have mild, moderate, severe,chronic, acute, etc. instances. For example, as shown in FIG. 4A, in oneillustrative embodiment, this lexicon consisted of over 11,000 uniqueterms covering the space of 78 core findings and 9 modifier types whichrepresents the largest set of core finding labels assembled for chestradiographs to date.

The resulting core findings lexicon or vocabulary developed through anautomated or semi-automated process using the core findings lexicondevelopment computing tool provides a catalog of core finding labelsalong with their variants which can now be used to locate these corefindings in medical imaging reports, such as radiology reports, forimage labeling purposes. In one illustrative embodiment, the corefinding lexicon describes the following columns: (a) the core findingterm; (b) its synonyms which include alternate ways of referring to thecore finding, visually similar equivalents, and spelling error variantsdue to spoken word translations; (c) the category of the core findingsuch as tubes and lines finding, devices, diseases, etc.; (d) theontological relationship to another higher level term describing a groupof core findings, e.g., fracture is an ontological group for corefindings such as sternum fracture, spine fracture, etc.; (e) concept IDas an identifier to place the term in the overall lexicon; (f) theanatomical region where the finding occurs; (g) source of vocabulary(UMLS or other), (h) coding system for the concept ID (ICD9, 10 orinternal coding called cxr). An example of a portion of a core findinglexicon in accordance with this illustrative embodiment is shown in FIG.4B.

In accordance with the illustrative embodiments of the presentinvention, this initial core finding lexicon is used as a basis forperforming fine-grained label generation. This fine-grained labelgeneration comprises four primary operations performed by correspondingcomputing tools specifically configured to perform these operations.These four primary operations consist of (a) core finding and modifierdetection, (b) phrasal grouping, (c) negation sense detection, and (d)fine-grained finding pattern completion.

Detecting Core Findings in Reports

With regard to detecting core findings in medical imaging reports, theillustrative embodiments use a lexicon or vocabulary driven conceptextraction process to identify all occurrences of core findings and/ortheir synonym variants in sentences within medical imaging reports. Themedical imaging reports, e.g., radiology reports, are pre-processed toisolate the sections describing the findings and impression. Often,these are indicated by section headings found in medical imaging reportsand thus, the pre-processing can use natural language processing toidentify section headings and the terms in such section headings thatare indicative of findings or impressions. The lexicon or vocabularydriven extraction process is then executed on the identified sections ofthe medical imaging reports.

In order to perform the lexicon or vocabulary driven extraction process,the process first builds a vocabulary index data structure in which eachsynonym of the core finding points to the core finding phrase in thelexicon. This index may be built upon the core findings lexicon orvocabulary developed through the automated or semi-automated processusing the core findings lexicon development computing tool discussedpreviously. This ensures that a match to a core finding phrase can befound through its synonyms using the vocabulary index data structure. Toensure a match to various word forms of the core finding phrases, thecore finding terms are pre-processed by retaining essential prefixes ofterms within a core findings prefix data structure such that thecombined presence of these prefixes points to the actual core findingphrase in the vocabulary (lexicon). For example, in FIG. 5, column 510lists the prefix strings for the core findings phrases in column 520.Matching sentences in a textual report for each of the prefix stringsare shown in column 530.

The set of prefixes that best discriminate a core finding phrase (alsoreferred to as a vocabulary phrase) can be determined by a deterministicalgorithm that iteratively shortens each term in a phrase until it failsto be discriminatory in identifying the vocabulary phrase. An example ofsuch a deterministic algorithm that identifies the smallestdistinguishable prefix per term in a phrase is shown in FIG. 6A. In FIG.6B an example of a longest common subfix (LCF) algorithm is provided, aswill be discussed hereafter.

In one illustrative embodiment, the core findings lexicon or vocabularyis pre-processed by this smallest prefix building algorithm to recordall prefix strings in the vocabulary index. Generation of the prefixstrings is part of the preparation to put the lexicon in an index. Theprefix generation process reduces the chance of false matches whileincreasing precision since the prefix generated is relatively unique forthe vocabulary term. For detecting the vocabulary phrase, all prefixterms from vocabulary phrases are searched within the portions ofnatural language content, e.g., sentences, from the relevant sections ofmedical imaging reports, e.g., the findings and impression sections, andthose vocabulary phrases with full matches to the prefixes are retained.This minimizes the false positives in matching the concepts,particularly for multi-term phrases. Once the candidate vocabularyphrases are identified, a detailed match is initiated within theportions of natural language content, e.g., sentences, in which theywere found using a dynamic programming algorithm to align the words ofcandidate vocabulary phrases to the portion of natural language content(hereafter assumed to be sentences, but which can be any multi-termportion of natural language content) using the prefixes. The resultingalignment guarantees the largest number of words of the vocabularyphrase are matched to the largest possible extend in the sentence whilestill maintaining the word order and allowing missed and spurious wordsin between.

For example, given a query vocabulary phrase S=<s₁s₂ . . . s_(k)> of Kwords and a candidate sentence T=<t₁t₂ . . . t_(N)> of N words, alongest common subfix (LCF) is defined as LCF(S,T)=<p₁p₂ . . . p_(L)>,where L is the largest subset of words from S that found a partial matchin T, and pi is a partial match of a word s₁ϵS to a word in T. A words_(i) in S is said to partially match a word t_(j) in T if it shares amaximum length common prefix p_(i) such that

${\frac{p_{i}}{\max\left\{ {{s_{i}},{t_{j}}} \right\}} \geq \tau},$

where τ is a threshold such that if the threshold is set to 1.0, theevaluation reduces to a case of finding exact matches to words of S.Aligning to prefixes was selected in order to correspond to the Englishgrammar rules where many word forms of words share common prefixes. Thisallows for the modeling of word variants, such as “regurgitated”,“regurgitating”, and “regurgitation”, as they all share a sufficientlylong prefix “regurgitat.” The alignment to prefixes also allows formodeling spelling errors, particularly those that are made in the laterportion of a word which will be deemphasized during alignment.

As noted above, an example LCF based algorithm is shown in FIG. 6B. Inthe depicted LCF based algorithm, p_(max)(i, j) is the longest prefix ofthe strings s_(i)t_(j) and S is a mismatch penalty, which controls theseparation between matched words and prevents words that are too farapart in a sentence from being associated with the same vocabularyphrase, thus minimizing the effect of incorrect anaphora resolution in asentence. Using such an LCF based algorithm, a vocabulary phrase S issaid to be detected in a sentence T if

$\frac{{{LCF}\left( {S,T} \right)}}{S} \geq \Gamma$

for a threshold Γ. The choice of τ and Γ affect precision and recall inmatching and can be suitably chosen to meet specified criteria forprecision and recall based on a Receiver Operating Characteristic (ROC)curve analysis. It should be noted that the normalization in theprevious equation is on the length of the vocabulary phrase and not thesentence allowing matches to be found in long sentences.

Referring again to FIG. 5, the depicted table illustrates examples ofprefix extraction for terms within a vocabulary phrase to increasespecificity of matching. In FIG. 5, column 520 shows the vocabularyphrases that were recognized from sentences shown in column 530. As canbe seen, the LCF based algorithm, such as the one shown in FIG. 6B, isable to spot the occurrence of both “aortic sclerosis” and “aorticstenosis” in the sentence, even though the words “aortic” and “stenosis”are separated by several words in between. Similarly, the vocabularyphrase “left atrial dilatation” was matched to “Left Atrium: Left atrialsize is mildly dilated” even without a deep understanding of thelinguistic origins of the underlying words.

Associating Modifiers with Relevant Core Findings

The above vocabulary-driven phrasal detection algorithm can be appliedto the vocabulary of both core findings and modifiers in the corefindings lexicon (vocabulary) to appropriately tag phrases withinsentences. The first step in fine-grained finding detection is to detectthe core finding itself using the vocabulary-driven concept extractionmethod. This method also identifies other terms corresponding tomodifiers as well, such as anatomy, location, laterality, etc. Again,during lexicon development, both core findings and modifier types, withcorresponding modifier instances, are identified through thesemi-automated process, which can then be used to identify phrases innatural language content having core findings and modifiers. Thesubsequent steps perform natural language parsing, phrasal grouping,etc. By “tagging” what is meant is the identifying of the vocabularyterms from the lexicon within the sentence and marking them as such,i.e. marking them as core finding, modifier type, etc.

To generate fine-grained finding labels (FFLs), the modifiers areassociated with the relevant core findings. Doing this without fullnatural language understanding can be difficult. For example, in thesentence “The lungs are normally inflated without evidence of focalairspace disease, pleural effusion or pneumothorax” is the modifier“focal” associated with airspace disease only, or also with pleuraleffusion and pneumothorax?

The illustrative embodiments use a natural language parser, such as theEnglish Slot Grammar (ESG) parser, for example, which performs wordtokenization, sentence segmentation, morpho-lexical analysis, andsyntactic analysis to produce a dependency parse tree, which in the ESGparser mechanism is called the Slot Grammar (SG) parse tree. Using ESGand the SG parse tree as an example, in the SG parse tree, each treenode N is centered on a head term, which is surrounded by its left andright modifiers, which are, in turn, tree nodes. Each modifier M of Nfills a slot in N. The slot shows the grammatical role of M in N and isindicated by a tuple T=(t1, t2, . . . tk) which means that t1 is a termgrammatically related to modifiers t2, . . . tk. Here, an unknownmodifier is indicated by the symbol “u”. A sample SG parse tree for thesentence “The lungs are normally inflated without evidence of focalairspace disease pleural effusion or pneumothorax” is shown in FIG. 7A.The association tuples are also shown in FIG. 7A, such as for the word“without”, the tuple (6,5,7) indicates the word “without” is relatingthe term “inflate” to “evidence.” Associations that logically gotogether, such as adjectives describing nouns, are already indicated bythe ESG parser through numeric codes exceeding 100, such as for the term“pleural effusion” which has the slot structure (211) and is also seenby the pairing (12, 13).

Given such a dependency parse tree G and the tuples T_(G)=<T₁, T₂, . . .T_(N)> corresponding to the N tree nodes, where T_(i)=(t_(i1), . . .t_(ki)) is the tuple per node, a phrasal group is defined as P₁=(e₁, e₂,. . . e_(M)) where e_(j)=t_(jk)ϵT_(j) is the kth element of a tupleT_(j) and ∀_(j=1) ^(M)T_(j)∩T_(j+1)≠0. In other words, a phrasal groupis a connected component formed from the transitive closure of thetuples such that they have at least one element in common. Consider thesentence “Clear lungs without evidence of pneumonia”. The naturallanguage parser would produce a dependency parse tree like that shown inTable 1 below. In this case, it can be seen that (1,2,u) for “clear”indicates words 1 and 2 can be grouped together because they alreadyoccur in the dependency parse tree and are indicated by the parser. Theword “of1” (5,4,6) is similarly indicating that words 4, 5, and 6 belongto a connected component. Now the word “evidence2” (4,2,u) can be usedto infer that the words “evidence of pneumonia” can be further groupedwith “lung” (which in turn can be grouped with “clear”) to form a largerconnected component. Thus, the cues inside the dependency parse tree areused to recursively group words or terms into larger and largerconnected components. Initially each connected component may be a singleword or a few words already in a relationship such as “clear1” (1,2,u),but after the grouping algorithm we get a larger group (1,2,4,5,6) allin one phrasal group or (clear, lung, evidence, of, pneumonia).

TABLE 1 Example dependency parse tree for sentence “Clear lungs withoutevidence of pneumonia” .− nadj clear1 (1, 2, u) adj ϵ .−+ − subj(n)lung1 (2, u) noun |{grave over ( )}− nadjp without2 (3, u) adv r o−−−top evidence2 (4, 2, u) verb {grave over ( )}−−− vprep of1 (5, 4, 6)prep {grave over ( )}− objprep(n) pneumonia (6, u, u) noun

FIGS. 7A and 7B illustrate another example of the phrasal groupingprocess and the groups produced for the sentence shown at the top of thefigure. FIG. 7A shows a dependency parse tree of the sentence “The lungsare normally inflated without evidence of focal airspace disease pleuraleffusion or pneumothorax” generated by an ESG parser. FIG. 7B is adepiction of a phrasal grouping process in accordance with theillustrative embodiments, using a connected component analysis. In FIG.7B, the core findings from the core findings lexicon that occur withinphrasal groups are identified as elements 710-740. Core findings thatcross phrasal groups are identified as elements 750-760. The modifier isindicated as element 770. The term “Lung” in the depicted example isanother indicated modifier 780 of a “anatomy” type.

Since the core findings and modifiers were detected from a prior stageof processing, i.e. the first stage of the fine-grained finding labelswhere detection of the vocabulary terms of the lexicon, including termscorresponding to core findings and terms corresponding to modifiertypes, within the natural language content, these core findings andmodifiers are mapped back into the phrasal group by identifying phrasalgroups that contain core findings and/or modifiers of core findings inthe core findings lexicon or vocabulary. Phrasal groups that contain oneor more core findings are called “core phrasal groups” or “core groups”while the rest of the groups are called the “helper phrasal groups” or“helper groups”. In the depicted example, phrasal groups 1, 4, 5, and 6are core phrasal groups whereas the other groupings are helper groups.If a core finding is detected across two or more adjacent core groups,where adjacent core groups are groups in the parse tree that have anedge that directly connects the two groups such that adjacency is basedon the nearest consecutive words in the groups, then they are alsomerged to form a single core group as shown in FIG. 7B where theoriginal phrasal groups for “airspace” and “disease” are combined togenerate grouping 4. All modifiers present in helper groups areassociated with the core findings of their adjacent groups. Thus, inFIG. 7B, the modifier “focal” in helper group 3 is associated with thecore findings of the adjacent core group 4, i.e. “airspace disease”.FIG. 7B also lists the various phrasal groups and the two core findingassociations found in the sentence (shown as arcs).

Negated Instance Detection of Core Findings

To determine if a core finding is a positive or negative finding (e.g.,“no pneumothorax”), such that the correct value for a correspondingpositivity characteristic in the fine-grained label descriptor datastructure may be set, a two-step process is followed that combineslanguage structuring and vocabulary-based negation detection. Thelanguage structuring approach to negation detection starts from adependency parse tree of a sentence. A set of known negation dependencypatterns, such as may be developed by computerized natural languageprocessing (NLP) mechanism developers, is used to search for negationkeywords and the scope of words spanned by a negation keyword. Thenegation pattern detection algorithm iteratively identifies words withinthe scope of negation of a detected negation keyword based on dependencyparsing and pattern matching of the predetermined negation dependencypatterns. For example, let S be the set of negated words. The algorithmstarts by adding a collection of manually curated negation keywords orcues (e.g., “no”) into S, and then iteratively expanding S throughtraversing the dependency parse tree of a sentence until S becomesstable, i.e. no more words/terms are added to the set of negated wordsS.

FIG. 8 shows an example of negation detection for the sentence “There isno evidence suggesting that he has cancer.” Based on the computerizednatural language processing of the natural language content, e.g., thesentence shown in FIG. 8, and the negation pattern matching, thenegation scope, i.e. the set of negated words S, is determined to be“evidence”, “suggesting”, “has”, and “cancer”, and the target vocabularyphrase is identified as “cancer.”

The above described negation detection algorithm is dependent on thecorrectness of the dependency parse tree data structure. To ensure thatthe negation keywords, are being associated with the relevant corephrasal group, a vocabulary of “negation prior” and “negation post”terms is developed and utilized such that their occurrence prior or postthe core finding in the natural language content is a further indicationof negation or avoiding spurious negation detection. This negation priorand negation post evaluation may be performed after the languageanalysis of the negation detection algorithm operates on the parse treedata structure to identify patterns of negation. By explicitly lookingfor these negation terms indicating pre or post terms surrounding a corefinding, the negation detection can have improved precision. That is,the natural language processing of the negation detection algorithm thatidentifies patterns within the dependency parse tree uses the dependencyparse tree but does not explicitly account for the fact that it is thecore finding whose negated instance that is trying to be detected. Theuse of the pre and post negation terms reduced the negation detectionerror, such as from approximately 7% to approximately 2%. The pre- andpost-negation terms may be documented in the core finding lexicon. Byadding the pre- and post-negation term detection mechanism to thenegation detection algorithm, based on the pre and post negation termsin the core finding lexicon, performance of the negation detector wasfound to be improved by a significant amount.

Fine-Grained Finding Descriptor Formation

Through the above processes, core findings in portions of naturallanguage content of medical imaging reports are identified and thephrasal groups associated with core findings are further identified soas to identify which modifiers are associated with the core findings.Whether or not a core finding is positively or negatively identified inthese portions of natural language content is further determined usingthe extended negation detection algorithm described previously whichincludes both negation pattern detection and pre- and post-negation termoccurrence identification. These identified characteristics of a corefinding in medical imaging reports are then combined to form afine-grained finding descriptor data structure that identifies afine-grained finding pattern which can be used to identify similaroccurrences of the fine-grained finding pattern in other medical imagingreports.

To form the fine-grained finding descriptor data structure, using thefine-grained finding descriptor format previously described above, i.e.the tuple defined as F_(i)=<T_(i)|N_(i)|C_(i)|M_(i)*>, the fine-grainedfinding descriptor formation process begins with the core finding C_(i)and the associated modifiers M_(i) discovered during the phrasalgrouping process discussed above. For each core finding C_(i), itsfinding type is retrieved from the core findings lexicon or vocabulary.Further, due to the a priori knowledge captured in the core findingslexicon or vocabulary for the associated anatomical locations offindings, the fine-grained findings descriptor can be augmented with theanatomical location even when these are not specified in the naturallanguage content of the medical imaging report itself. In addition, thename of the core finding may be ontologically rolled-up to the corefindings from the core finding lexicon. That is, in the core findinglexicon, the core finding name and all of its synonyms are specified. Inaddition, the fine-grained finding name may be rolled-up into the corefinding name. For example, if sternum fracture was a core finding in thecore finding lexicon, the ontology column of the core finding lexiconwill include “fracture” while the synonym column may include “sternalfracture”, “sternum bone abnormality”, etc. (see example in FIG. 4B asdiscussed previously).

The results of the extended negation detection algorithm, indicatingwhether or not the core finding is positively or negatively indicated inthe natural language content, and thus, positively or negativelyindicated by the fine-grained finding pattern defined by thefine-grained finding descriptor, may be used to set the value of thenegation attribute Ni in the fine-grained finding descriptor datastructure.

Thus, all of the components of the fine-grained finding descriptor datastructure are provided through the processes above and used to generatethe fine-grained finding descriptor data structure. This process isrepeated for each core finding in each portion of natural languagecontent processed to generate a database of fine-grained findingdescriptor data structures that are found in medical imagine reports.The resulting fine-grained finding descriptor data structures may thenbe filtered so as to only retain a subset of fine-grained findingdescriptor data structures that satisfy desired frequency thresholds.That is, a frequency threshold may be predetermined that indicates howmany times a fine-grained finding descriptor data structure must befound present in medical imaging reports in order for it to bemaintained in a final set of fine-grained finding descriptor datastructures of the database, e.g., 100 instances.

The resulting database of fine-grained finding descriptor datastructures can then be used to train machine learning computer models,such as deep learning computer models and the like, to find instances ofsimilar fine-grain finding patterns in other natural language content.The detection of the fine-grained finding patterns defined by thefine-grained finding descriptor data structures in other naturallanguage content may be used as a basis for performing other cognitivecomputing operations, such as medical image labeling or the like. Forexample, rather than training a machine learning computer model, deeplearning computing model (neural network), or the like, to performmedical image labeling, such models, automated computing tools, orneural networks may be trained to perform other types of automatedcognitive computing operations, one of which may be patient synopsisgeneration. With a patient synopsis embodiment, the trained machinelearning computer model, deep learning computing model, or other trainedcomputing tool takes patient electronic medical records, which mayinclude medical imaging reports, and summarize the patient's medicalcondition based on the detection of fine-grained findings in thepatient's electronic medical records. Such a use will present thesynopsis to the medical practitioner who can then review the patient'selectronic medical record with a focused approach directed to theportions associated with the patient synopsis, e.g., locating theparticular medical images that would show the fine grained findingsindicated in the patient synopsis, identifying the lab results thatwould support/refute the fine grained findings, etc.

FIG. 7B lists fine grain finding descriptors, or fine grain findinglabels, extracted from the sentence shown in that figure. As can beseen, both positive and negative instances of findings have beenextracted by the process of the illustrative embodiments. FIG. 9provides a listing of examples of types of fine-grained findingdescriptors or labels (FFLs) 920 extracted from sentences 910 fromredacted medical imaging reports. The semantics column 930 shows themeaning of the FFL pattern shown in corresponding rows of column 920.That is, each FFL pattern in column 920 is of the formF_(i)=<T_(i)|N_(i)|C_(i)|M_(i)*>, as described previously. Thus there isa designated position for each modifier type. From the FFL patternextraction process, a unique FFL pattern is detected in the sentences ofthe natural language content of the medical imaging reports, describedin the above syntax, with the semantics indicated in column 930. A labelcode, such as L1, may be assigned to the unique FFL pattern to designateother patterns in other natural language content that correspond to theunique FFL and which can be referred to during machine learning.

As can be seen, important details of the finding are adequately capturedin the generated fine-grained finding descriptor or label (FFL) 920,despite the redaction such that the mechanisms of the illustrativeembodiments may be run on redacted medical imaging reports and yet stillgenerate a sufficiently detailed fine-grained finding descriptor orlabel to be used to trained machine learning computer models. In oneillustrative embodiment, by mining the findings and impression sectionsof over 220,000 radiology reports, the above process of the illustrativeembodiments was able to record all possible fine-grained findingdescriptors/labels that could be extracted and then, by retaining onlythose fine-grained finding descriptors/labels that were found in atleast 100 medical imaging reports, a total of 457 fine-grained findinglabels were selected. Of these, 78 were the original core labelsidentified in the core finding lexicon, and the remaining werefiner-grained labels with modifiers extracted automatically using theabove processes. FIG. 9 provides an example of some fine-grained findinglabels extracted from medical imaging reports and retained as part of afine grained finding descriptor database using the processes of theillustrative embodiments.

Training Machine Learning Computer Models for Image Labeling

Having developed a database of fine-grained finding descriptor datastructures, which define fine grained finding labels (FFLs) ordescriptors, the database may be used for various downstream artificialintelligence and cognitive computing operations. These artificialintelligence and cognitive computing operations may involve trainedmachine learning/deep learning models or may involve other computerlogic that implements complex analysis and evaluation of data structuresfor presentation of information otherwise not able to be easilyidentifiable by human users or to perform computer operations thatcannot be practically performed by human beings due to various factorsincluding, but not limited to, the volume of data being evaluated, thecomplexity of relationships between data that must be evaluated, or thelike.

In some illustrative embodiments, the FFLs defined in the fine-grainedfinding descriptor data structures may be used in downstream computingsystems to perform operations such as identifying an highlighting orotherwise accentuating portions of electronic medical records and/orsummarizations of electronic medical records that have a matching FFL.This will provide additional focus to medical imaging subject matterexperts on portions of complex electronic medical records/summarizationson the most important portions corresponding to findings which mayaffect a patient's diagnosis, treatment, or other understanding of thehealth condition of the patient. In such a downstream computing system,a machine learning model may be utilized, or may not be utilized. Thatis a computerized pattern matching mechanisms may be employed which doesnot require machine learning to operate, yet provides a complex analysisof electronic medical record content using other computer constructs,such as a rules engine or the like.

However, in other illustrative embodiments, the FFLs of the fine-grainedfinding descriptor data structures in the database may be implemented totrain a machine learning/deep learning (ML/DL) computer model that isable to distinguish between the fine-grained finding labels. As notedabove, while the illustrative embodiments may be used to train ML/DLcomputer models for identifying FFLs in natural language content tofacilitate various types of cognitive computing operations, oneprinciple cognitive computing operation for which such a ML/DL computermodel may be trained is to perform medical image labeling, i.e.identifying structures, abnormalities, etc. associated with findings inmedical images and appropriately labeling them as such. Such learninginvolves correlating features extracted from medical images withfindings found in the natural language content of corresponding medicalimaging reports such that the trained ML/DL computer model learnsassociations and patterns between medical image features and findingsspecified in the natural language content. Through training of the ML/DLcomputer model, these associations are learned and can be used toidentify similar patterns in other inputs of medical images and/ormedical imaging reports. For example, given features, e.g., an imagepattern, extracted from a medical image, the trained ML/DL computermodel may predict the labels for the extracted features based on thelearned associations with FFLs defined by the fine-grained findingdescriptor data structures. Similarly, given a medical imaging report,and identifying an instance of natural language content matching a FFLof a fine-grained finding descriptor data structure, the trained ML/DLcomputer model can predict the location in a medical image of acorresponding structure, abnormality, etc. based on the learnedassociations of the FFL of the fine-grained finding descriptor datastructure with medical image features.

The learning of FFLs from chest radiographic images, for example, is afine-grained classification problem for which single networks used forcomputer vision problems may not yield the best performance as largetraining sets are still difficult to obtain. Concatenating differentimage dataset pretrained features from different trained ML/DL computermodels, e.g., neural networks, can improve classification. Thus, in someillustrative embodiments, pretrained features, such asImageNet-pretrained features, from different trained ML/DL computermodels for computer vision are combined through a feature pyramidnetwork using features across multiple scales. An example of a ML/DLcomputer model of this type using concatenation of different imagedataset pretrained features is shown in FIG. 10.

For this example embodiment in FIG. 10, the VGGNet and ResNet are usedas feature extractors and their lower-level features are retained. Inparticular, in one illustrative embodiment, from the VGGNet, featuremaps with 128, 256, and 512 feature channels are used, which areconcatenated with the feature maps from the ResNet of the same spatialsizes which have 256, 512, and 1024 feature channels. Dilated blocks areused to learn the high-level features from the extracted features, e.g.,ImageNet features. Each dilated block is composed of dilatedconvolutions for multi-scale features, and uses a skip connection ofidentity mapping to improve convergence and spatial dropout to reduceoverfitting. Group normalization (e.g., 16 groups) is also used withRectified Linear Unit (ReLU). Dilated blocks with different featurechannels are cascaded with max pooling to learn more abstract features.

Second-order pooling is used, which is proven to be effective forfine-grained classification and maps the features to ahigher-dimensional space where they are more separable. In someillustrative embodiments, the second-order pooling is implemented as a1×1 convolution followed by global square pooling.

Image augmentation with rigid transformations is used to avoidoverfitting. As most of an image should be included, in someillustrative embodiments, the augmentation is limited to rotation(+/−10°) and shifting (+/−10°). In one illustrative embodiment, theprobability of an image to be transformed is 80% and the optimizer Nadamis used with a learning rate of 2×10⁻⁶, a batch size of 48, and 20epochs. In some illustrative embodiments, such as in the illustrativeembodiments described hereafter with regard to automated imaging reportgeneration, to ensure efficient machine learning, two instances of theML/DL computer model shown in FIG. 10 are trained, one for core findinglabels (CFL labels) and the other for the detailed fine-grained findinglabels (FFL labels) which have support of at least a predeterminednumber of images, e.g., 100 images, for training to exploit the mutuallyreinforcing nature of the coarse-fine labels. Due to the variability inthe size of the dataset per FFL, the Area under the ROC Curve (AUC) perFFL is not always a good indicator for precision on a per image level asit is dominated by the negative examples. To ensure as few irrelevantfindings as possible while still detecting critical findings within animage, operating points on a ROC curve per label are selected based onoptimizing the F1 score, a well-known measure of accuracy, as

${L(\theta)} = {- {{\ln\left( {\frac{1}{n}{\sum\limits_{i = 1}^{n}\;{F\; 1_{i}(\theta)}}} \right)}.}}$

In one illustrative embodiment, a deep neural network architecture wasdesigned that combines the advantages of pretrained features with amulti-resolution image analysis through a feature pyramid network forfine grained classification. Specifically VGGNet²¹(16 layers) and ResNet(50 layers) were used as the initial feature extractors, which weretrained on multi-million images from ImageNet. Dilated blocks composedof multi-scale features and skip connections were used to improveconvergence while spatial dropout was used to reduce overfitting. Groupnormalization (16 groups) was used, along with Rectified Linear Unit(ReLU) as activation function. Dilated blocks with different featurechannels were cascaded with max pooling to learn more abstract features.Bilinear pooling was used for effective fine-grained classification.

To train the deep learning model, the modeling dataset was split intothree partitions for training, validation and testing. Since existingmethods of random splitting cannot ensure adequate number of images forlow incidence label training, the splitting algorithm in this exampleembodiment sorted the labels by their frequencies of occurrences. Thesplitting algorithm then iteratively assigned the images from distinctpatients to the three partitions in the ratio of 70-10-20% for training,validation and testing. Once the number of patients in each split wasdetermined per label, the assignment of the patients/images was stillrandom. Thus, the algorithm ensured that the prevalence distributionswere similar for training, validation and testing partitions whileminimizing the selection bias through random sampling of images.

The deep learning model was trained on all finding labels (CFLs and FFLsdepending on the model trained). As the images were of high resolution(e.g., 1024×1024), training took approximately 10 days. The Nadamoptimizer was used for fast convergence with the learning rate as2×10⁻⁶. Two NVIDIA Tesla V100 GPUs with 16 GB memory were used formulti-GPU training with a batch size of 12 and 30 epochs.

Computing Environment and Computing Architecture

The illustrative embodiments provide an improved computing tool andimproved computing tool methodology to automatically learn fine-grainedfinding labels (FFLs) used in the natural language content of medicalimaging reports and generate fine-grained finding descriptor datastructures that define fine-grained finding patterns. The fine-grainedfinding descriptor data structures can then be used to train machinelearning/deep learning (ML/DL) computer models, such as neural networksor the like, to perform artificial intelligence (cognitive computing)operations based on the detection of such fine-grained finding patternsin other natural language content, such as other medical imagingreports, other portions of patient electronic medical records, or thelike. In this way, improved automated computing tools are provided toassist human medical practitioners in understanding and identifyingfindings in a patient's electronic medical records (EMRs), therebyimproving the way that the human medical practitioner can perform theirduties of providing care to their patients. That is, the improvedautomated computing tools are able to surface, from the largecombination of medical information data of a patient's electronicmedical record, the subset of information of particular importance forthe medical practitioner's attention corresponding to fine-grainedfindings. This reduces the likelihood that the medical practitioner willmiss information in the patient's EMR, or miss associations ofinformation in the patient's EMR because this information is obscured bythe complexity and/or volume of information present in the patient EMR,or the difficulty in identifying specific structures/abnormalities inmedical imaging data. The improved computing tools of the illustrativeembodiments automatically learn fine-grained finding patterns andautomatically uses the learned fine-grained finding patterns to identifyinstances of such patterns in patient electronic medical records tothereby extract associated information from the patient electronicmedical records and perform other artificial intelligence (cognitivecomputing) based operations to assist medical practitioners, such asautomatically labeling structures/abnormalities in medical images,automatically generating preliminary medical imaging reports,automatically generating patient electronic medical record summariesthat specify specific subsets of pertinent information extracted fromthe patient electronic medical record that is of particular importanceto medical practitioner review, etc.

As the present invention is specifically directed to improved automatedcomputing tools and automated computing tool methodologies, it can beappreciated that the illustrative embodiments may be utilized in manydifferent types of data processing environments in which one or morecomputing devices are specifically configured through software/hardwarelogic to perform the specific automated computing tool processespreviously described above. In order to provide a context for thedescription of the specific elements and functionality of theillustrative embodiments, FIGS. 11 and 12 are provided hereafter asexample environments in which aspects of the illustrative embodimentsmay be implemented. It should be appreciated that FIGS. 11 and 12 areonly examples and are not intended to assert or imply any limitationwith regard to the environments in which aspects or embodiments of thepresent invention may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

FIG. 11 depicts a pictorial representation of an example distributeddata processing system in which aspects of the illustrative embodimentsmay be implemented. Distributed data processing system 1100 may includea network of computers in which aspects of the illustrative embodimentsmay be implemented. The distributed data processing system 1100 containsat least one network 1102, which is the medium used to providecommunication links between various devices and computers connectedtogether within distributed data processing system 1100. The network1102 may include connections, such as wire, wireless communicationlinks, or fiber optic cables.

In the depicted example, servers 1104A-D are connected to network 1102along with network attached storage unit 1108. In addition, clientcomputing devices 1110, 1112, and 1114 are also connected to network1102. These client computing devices 1110, 1112, and 1114 may be, forexample, personal computers, network computers, proprietary servers, orthe like. In the depicted example, one or more of the servers 1104A-Dprovides data, such as boot files, operating system images, and/orapplications to the client computing devices (clients) 1110, 1112, and1114. Client computing devices 1110, 1112, and 1114 are clients toservers 1104A-D in the depicted example. Distributed data processingsystem 1100 may include additional servers, clients, and other devicesnot shown.

In the depicted example, distributed data processing system 1100 is theInternet with network 1102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers, consisting of thousands of commercial, governmental,educational and other computer systems that route data and messages. Ofcourse, the distributed data processing system 100 may also beimplemented to include a number of different types of networks, such asfor example, an intranet, a local area network (LAN), a wide areanetwork (WAN), or the like. As stated above, FIG. 11 is intended as anexample, not as an architectural limitation for different embodiments ofthe present invention, and therefore, the particular elements shown inFIG. 11 should not be considered limiting with regard to theenvironments in which the illustrative embodiments of the presentinvention may be implemented.

As shown in FIG. 11, one or more of the computing devices, e.g., server1104A, may be specifically configured to implement a core findinglexicon development computing tool 1120, a fine-grained findingdescriptor generation computing tool 1130, a machine learning/deeplearning (ML/DL) computer model training computing tool 1140 inaccordance with one or more of the illustrative embodiments describedherein. The configuring of the computing device(s) may comprise theproviding of application specific hardware, firmware, or the like tofacilitate the performance of the operations and generation of theoutputs described herein with regard to the illustrative embodiments.The configuring of the computing device(s) may also, or alternatively,comprise the providing of software applications stored in one or morestorage devices and loaded into memory of the computing device(s), suchas server 1104A, for causing one or more hardware processors of thecomputing device to execute the software applications that specificallyconfigure the processors to perform the operations and generate theoutputs described herein with regard to the illustrative embodiments.Moreover, any combination of application specific hardware, firmware,software applications executed on hardware, or the like, may be usedwithout departing from the spirit and scope of the illustrativeembodiments. In this way, the computing device(s) configured to performthe computer specific operations of the present invention arespecialized computing devices performing computer operations based oncomputer specific logical structures in a manner that cannot bepractically performed manually or through human mental processes.

That is, it should be appreciated that once the computing device(s)is/are configured in one of these ways, the computing device becomes aspecialized computing device specifically configured to implement themechanisms of the illustrative embodiments and is not a general purposecomputing device. Moreover, as described herein, the implementation ofthe mechanisms of the illustrative embodiments improves thefunctionality of the computing device and provides a useful and concreteresult that facilitates a computer specific automated learning offine-grained finding labels used in medical imaging reports and theautomated training of machine learning/deep learning computer models toperform artificial intelligence (cognitive computing) operations basedon the automatically learned fine-grained finding labels.

The core finding lexicon development computing tool 1120 is specificallyconfigured to perform the operations described previously (see thesection of the description above entitled “Fine-Grained FindingDescriptor and Core Finding Vocabulary”), either automatically orsemi-automatically, to perform core findings and modifier detection. Thecore finding lexicon development computing tool 1120 may operateautomatically or semi-automatically to process a corpus 1122 of medicalimaging reports and medical imaging data to identify core findingsterms/phrases in these medical and a core set of modifier types, suchthat these core findings terms/phrases may be used to generate aninitial core findings lexicon/vocabulary. In one illustrativeembodiment, the core finding lexicon development computing toolidentifies all of the instances of core findings in the electronicdocuments, e.g., medical imaging reports and corresponding medical imagedata, in the corpus 1122 and presents these core findings to subjectmatter experts (SMEs) for evaluation as to whether or not the corefinding should be maintained as part of the lexicon.

As discussed above, the core finding lexicon development computing tool1120 uses a vocabulary-driven concept extraction algorithm to spot alloccurrences of core concepts and/or their variants, e.g., synonyms,misspellings, alternative forms, etc., in an electronic corpus ofelectronically stored medical imaging reports. For example, using achest X-ray embodiment, the vocabulary-driven concept extractionalgorithm is used to create a core finding lexicon or vocabular tocatalog all possible findings in medical images, such as chest x-rays,for example, which recorded the names, spelling variants, synonyms, etc.for core findings and modifiers by analyzing a large set ofelectronically stored medical imaging reports, e.g., 200,000 medicalimaging reports. The core finding lexicon development computing tool1120 generates an initial core finding lexicon or vocabulary datastructure 1125 that specifies the core findings and their correspondingfinding types and initial set of modifier types, and correspondingmodifier instances (see FIG. 4). This core finding lexicon datastructure 1125 may then be used by the fine-grained finding descriptorgeneration computing tool 1130 to identify instances of core findings inmedical imaging report data structures of a corpus of such medicalimaging report data structures, and generate fine-grained findingdescriptor data structures based on the identified instances.

The fine-grained finding descriptor generation computing tool 1130includes a core finding and modifier detector 1132, a phrasal groupingengine 1134, a negation sense detector 1136, and a fine-grained findingdescriptor generator 1138. The core finding and modifier detector 1132uses the core finding lexicon data structure 1125 and avocabulary-driven concept extraction algorithm to identify occurrencesof core concepts and their variants in natural language content of acorpus of medical imaging reports 1150, which may be the same,different, or overlapping corpus of medical imaging reports as the oneused for lexicon development 1122. The fine-grained finding descriptorgeneration computing tool 1130, in some illustrative embodiments, uses alexicon pre-processor 1131 implementing a smallest prefix buildingalgorithm to pre-process the core finding lexicon (vocabulary) datastructure 1125 to ensure high precision. The lexicon pre-processor 1131uses a dynamic programming algorithm to align the words of candidatevocabulary phrases to portions of natural language content in themedical imaging reports 1150 using the smallest prefixes with theresulting alignment guaranteeing the largest number of words of thevocabulary phrase being matched to the largest possible extent in theportion of natural language content while still maintaining the wordorder and allowing missed and spurious words in-between, as describedpreviously. In order to ensure high recall, the vocabulary-drivenconcept extraction algorithm of the core finding and modifier detector1132 uses a longest common subfix (LCF) algorithm to perform anapproximate match to a target vocabulary phrase in the pre-processedcore finding lexicon data structure 1133 within a portion of naturallanguage content of a medical imaging report 1150. In this way, phrasesin the natural language that are believed to contain core findingsand/or modifiers may be identified.

The phrasal grouping engine 1134 uses a natural language processing(NLP) parser, such as an English Slot Grammar (ESG) parser in someillustrative embodiments, to parse the natural language content of amedical imaging report 1150 to generate a dependency parse tree. Thephrasal grouping engine 1134 operates on the dependency parse tree toperform connected component clustering based on the placement of termsin the parse tree, e.g., based on a slot grammar placement of the terms.Core findings and modifiers are then identified within each grouping andassociated with each other or with adjacent groups, as previouslydescribed above. In this way, the phrasal grouping engine 1134identifies instances of core findings and corresponding modifiers inmedical imaging reports which can be used to create the fine-grainedfinding descriptor data structures.

The negation sense detector 1136 performs the operations describedpreviously for detecting negation of core findings in the naturallanguage content of the medical imaging report. For example, in someillustrative embodiments, a two-step process is utilized that combineslanguage structuring and vocabulary-based negation detection. Thelanguage structuring based negation detection starts from the dependencyparse tree of the natural language content and looks for knowndependency patterns corresponding to negation, as specified in apredefined set of known dependency patterns and using pattern matchingto find matching patterns in the given dependency parse tree. In thisway, negation keywords are identified in the dependency parse tree andthe scope of words encompassed by these negation keywords is identifiedby the known negation dependency patterns. The negation patterndetection algorithm iteratively identifies words within the scope ofnegation based on dependency parsing. To ensure that the negationmodifiers are being associated with the relevant core phrase, avocabulary of “negation prior” and “negation post” terms is also usedsuch that detection of their occurrence prior or post the core findingis used as a further indication of negation or avoiding spuriousnegation detection. Negation detected by the negation sense detector1136 is used to set a corresponding negation attribute in thefine-grained finding descriptor data structure.

The fine-grained finding descriptor generator 1138 generates thefine-grained finding descriptors corresponding to the instances of corefindings and associated modifiers found in the various medical imagingreports of the corpus of medical imaging reports 1150. As discussedpreviously, the illustrative embodiments utilize a new fine-grainedfinding descriptor data structure to define fine-grained findingpatterns found in natural language content of medical imaging reports.In some illustrative embodiments, this fine-grained finding descriptortakes the form of F_(i)=<T_(i)|N_(i)|C_(i)|M_(i)*> where again F_(i) isthe fine-grained label, T_(i) is the finding type, N_(i)=yes|no andindicates a positive or ruled-out finding, C_(i) is the core findingitself, and M_(i) are one or more of the possible finding modifiers.While this format is used in some of the illustrative embodiment, theillustrative embodiments are not limited to this format. Other forms andformats of descriptor data structures that associate core findings withmodifiers of the core findings may be used without departing from thespirit and scope of the present invention.

With the above format of a fine-grained finding descriptor as an exampleimplementation, the attributes, or fields, of the descriptor arepopulated with the resulting fine-grained finding information obtainedthrough the operation of the other elements 1132-1136 of thefine-grained finding descriptor generation computing tool 1130. That is,the core finding attribute C_(i) is populated with the core finding fromthe lexicon 1125 for which a match was found in a medical imaging reportof the corpus 1150 by the core finding and modifier detector 1132.Similarly, the core finding type T_(i) is populated with informationpresent in the lexicon 1125 specified through the lexicon 1125 buildingprocess implemented by the core finding lexicon development computingtool 1120, e.g., see first column in FIG. 2 and the category column inFIG. 4. The modifiers M_(i) are populated by the modifiers discoveredthrough the phrasal grouping operations performed by the phrasalgrouping engine 1134. The negation attribute Ni is populated with avalue corresponding to whether or not the core finding was determined,by the negation sense detector 1136, to be negatively indicated by othernatural language content in the medical imaging report 1150.

Thus, the fine-grained finding descriptor generator 1138 generates afine-grained finding descriptor data structure, e.g.,F_(i)=<T_(i)|N_(i)|C_(i)|M_(i)*>, for each instance of a core findingfound in each medical imaging report processed from the corpus 1150. Thegenerated fine-grained finding descriptor data structures may be storedtemporally for further evaluation as to whether or not they should bemaintained in a fine-grained finding descriptor database 1160 fortraining ML/DL computer models. The evaluation of whether or not tomaintain certain fine-grained finding descriptor data structures may bedetermined based on various automatically applied criteria applied bythe fine-grained finding descriptor generator 1138, and may include SMEreview in some illustrative embodiments. The automatically appliedcriteria, for example, may be a frequency of occurrence within thecorpus 1150 compared to a predetermined threshold, e.g., 100. That is,the number of instances of the negatively/positively indicated corefinding and modifiers specified in the fine-grained finding descriptordata structure being present within the corpus 1150 is calculated fromthe generated descriptors and the number of instances are compared tothe predetermined threshold value. If the number of instances equals orexceeds the threshold, then an instance of the fine-grained findingdescriptor data structure is maintained in the database 1160.

Alternatively, the fine-grained finding descriptor generator 1138 maynot generate and store a separate instance of the fine-grained findingdescriptor data structure for every instances of the samenegatively/positively indicated core finding and modifiers. To thecontrary, the fine-grained finding descriptor generator 1138 maygenerate the fine-grained finding descriptor data structure and compareit to previously generated fine-grained finding descriptor datastructure to determine if there is already a matching fine-grainedfinding descriptor that was generated. If there is a matchingfine-grained finding descriptor, then a counter associated with thematching fine-grained finding descriptor data structure is incremented.Thus, a single fine-grained finding descriptor data structure isgenerated for instances of each different fine-grained findingdescriptor found in the corpus 1150 with a counter being used tomaintain a count of how many instances of that fine-grained findingdescriptor were found to be present in the corpus 1150. This countervalue may then be used to compare to the predetermined threshold todetermine whether to maintain the fine-grained finding descriptor datastructure as part of the database 1160 or not.

As a result of the above processes of the fine-grained findingdescriptor generation computing tool 1130, a database 1160 offine-grained finding descriptor data structures is generated. Thefine-grained finding descriptors, or fine-grained finding labels (FFLs),represented in these data structures of the database 1160 may be used totrain ML/DL computer models for performing various types of artificialintelligence (cognitive computing) computer operations on new inputdata. That is, the database 1160 may be accessed by the ML/DL computermodel training computing tool 1140 in accordance with one or more of theillustrative embodiments described herein, to train a ML/DL computermodel for a specific purpose, such that the trained ML/DL computer model1170 applies its machine learned specialized training to evaluate newdata and provide useful results that are not able to be obtained throughgeneric computing operations, such as loads, stores, basic computermathematical operations, and the like. It should be appreciated that theresulting trained ML/DL computer model 1170 need not be executed on thesame computing device or devices on which the ML/DL computer modeltraining computing tool 1140 executes, and in fact the ML/DL computermodel training computing tool 1140 may also execute on a differentcomputing device from the core finding lexicon development computingtool 1120 and/or the fine-grained finding descriptor generationcomputing tool 1130. That is, each of the elements 1120-1170 may in factbe implemented on different computing devices in the computingenvironment.

In some illustrative embodiments, the ML/DL computer model trainingcomputing tool 1140 may train different instances of the ML/DL computermodel 1170 which are each separately deployed for runtime execution onthe same or different computing devices and/or may train a single ML/DLcomputer model which is then deployed to the same or different computingdevices as separate instances. Furthermore, in some embodiments, theML/DL computer model training computing tool 1140 may perform trainingof a ML/DL computer model remotely such that the ML/DL computer modelstays on a user's local computing device, but is trained through amachine learning process in which the ML/DL computer model trainingcomputing tool 1140 provides the inputs to the ML/DL computer model,receives the outputs from the ML/DL computer model, and adjustsoperational parameters of the ML/DL computer model to reduce loss/errorin the outputs of the ML/DL computer model.

As mentioned above, the ML/DL computer model may be trained to performvarious types of artificial intelligence (cognitive computing)operations. An example of one type of artificial intelligence operation,for which a ML/DL computer model may be trained based on thefine-grained finding descriptors or fine-grained finding labels (FFLs)in the database 1160, is medical image labeling. That is, the ML/DLcomputer model 1170 may be trained to take, as input, a medical imagedata structure, perform image analysis on the medical image datastructure, such as a pattern recognition operation on the medical image,and label structures, anomalies, and the like, in the medical image withfine-grained finding labels by matching the patterns found in themedical image with corresponding fine-grained finding descriptor datastructures in the database 1160. The information in the fine-grainedfinding descriptor data structures may be used to generate the actuallabels that are applied to the patterns in the medical image to therebygenerate fine-grained finding labeled medical image data which providesgreater insights into the internal medical condition of patients. Theresulting fine-grained finding labeled medical image data may be used asa basis for presentation of the medical images along with thefine-grained finding labels pinpointing the structures/anomalies in themedical image and the fine-grained findings corresponding to thosestructures/anomalies. Again, an example ML/DL computer model for medicalimage labeling using the database 1160 is shown in FIG. 10 and describedabove.

With regard to training the ML/DL computer model 1170, as discussedpreviously, each FFL pattern can be denoted by an label identifier whichcan be used to perform machine learning training of the ML/DL computermodel 1170, where the ML/DL computer model 1170 is given a trainingimage and its corresponding label vector indicating all the FFL patternspresent (1 if the image contains a particular FFL pattern and 0otherwise). The task of the machine learning is to learn a function thatmaps the extracted image features/patterns to those labels in the labelvector such that when similar image features/patterns are detected innon-labeled images, the trained ML/DL computer model 1170 is able to mapthose features/patterns to predicted FFL patterns and generateprobability values or scores indicating the likelihood that the FFLpattern applies to the input non-labeled image.

The trained ML/DL computer model 1170 may also be trained for variousother operations, such as patient medical condition synopsis or summarygeneration, for example. That is, the ML/DL computer model 1170 may betrained using the database 1160 to identify instances of the FFLsdefined by the descriptor data structures present in the database 1160in patient electronic medical records, which may include medical imagingreports as well as other electronically stored medical information fromvarious source computing systems, e.g., pharmacies, doctor offices,hospitals, medical laboratories, medical imaging companies, medicalsupply stores, etc. This patient medical information data may becompiled from the various source computing systems into one or moreelectronic medical records that may be processed by the trained ML/DLcomputer model 1170 or a plurality of trained ML/DL computer models, ofwhich the trained ML/DL computer model 1170 may be one, in order togenerate a summary of the relevant patient medical condition informationto be presented to a medical practitioner, such as based on a currentmedical condition of the patient, based on a specific query submitted bythe medical practitioner, or the like.

For example, the trained ML/DL computer model 1170 may be trained toclassify text in the patient electronic medical record by extractingfeatures from the text and matching them with the core findings andmodifiers specified in the FFLs defined by the fine-grained findingdescriptor data structures of the database 1160. In this case, there maybe a separate class associated with each FFL of each fine-grainedfinding descriptor data structure and the ML/DL computer model 1170 istrained, through a machine learning process, to evaluate featuresextracted from the text of the patient electronic medical records andpredict whether the pattern of features matches one or more of the FFLs.The fine-grained finding descriptor data structure(s) associated withthe classification prediction(s) made by the trained ML/DL computermodel 1170 may be used as a basis for composing a natural languagedescription of the findings as an indicator of a medical condition ofthe patient. For example, the matching portions of text from the patientelectronic medical record may be identified and provided as part of thepatient summary and the core finding, modifiers, and negation attributesof the fine-grained finding descriptor data structure may be used as abasis for composing a natural language representation of the FFL of thefine-grained finding descriptor data structure. This is just one exampleof one way in which a patient summary generation AI operation may beimplemented by a trained ML/DL computer model 1170 trained using thedatabase 1160 generated by the processes of the illustrativeembodiments.

As noted above, the mechanisms of the illustrative embodiments utilizespecifically configured computing devices, or data processing systems,to perform the operations for developing a core finding lexicon,generating fine-grained finding descriptors based on the core findinglexicon, and training a ML/DL computer model based on the FFLs definedin the fine-grained finding descriptors. These computing devices, ordata processing systems, may comprise various hardware elements whichare specifically configured, either through hardware configuration,software configuration, or a combination of hardware and softwareconfiguration, to implement one or more of the systems/subsystemsdescribed herein. FIG. 12 is a block diagram of just one example dataprocessing system in which aspects of the illustrative embodiments maybe implemented. It should be appreciated that while FIG. 12 may resembleother diagrams of data processing systems, the data processing systemsand computing devices implementing the illustrative embodiments are notgeneric computing devices. They are specialized computing devices thatare specifically configured to perform the non-generic computeroperations realizing the functions and operations described herein in anautomated or semi-automated manner. These operations described hereinare specific improved computer operations that can only be performed bya specialized data processing system, computing device, or computer toolthat is specifically configured to perform these operations which cannotbe practically performed within a human mind.

Data processing system 1200 is an example of a computer, such as server1104 in FIG. 11, in which computer usable code or instructionsimplementing the processes and aspects of the illustrative embodimentsof the present invention may be located and/or executed so as to achievethe operation, output, and external effects of the illustrativeembodiments as described herein. In the depicted example, dataprocessing system 1200 employs a hub architecture including north bridgeand memory controller hub (NB/MCH) 1202 and south bridge andinput/output (I/O) controller hub (SB/ICH) 1204. Processing unit 1206,main memory 1208, and graphics processor 1210 are connected to NB/MCH202. Graphics processor 1210 may be connected to NB/MCH 1202 through anaccelerated graphics port (AGP).

In the depicted example, local area network (LAN) adapter 1212 connectsto SB/ICH 1204. Audio adapter 1216, keyboard and mouse adapter 1220,modem 1222, read only memory (ROM) 1224, hard disk drive (HDD) 1226,CD-ROM drive 1230, universal serial bus (USB) ports and othercommunication ports 1232, and PCI/PCIe devices 1234 connect to SB/ICH1204 through bus 1238 and bus 1240. PCI/PCIe devices may include, forexample, Ethernet adapters, add-in cards, and PC cards for notebookcomputers. PCI uses a card bus controller, while PCIe does not. ROM 1224may be, for example, a flash basic input/output system (BIOS).

HDD 1226 and CD-ROM drive 1230 connect to SB/ICH 204 through bus 1240.HDD 1226 and CD-ROM drive 1230 may use, for example, an integrated driveelectronics (IDE) or serial advanced technology attachment (SATA)interface. Super I/O (SIO) device 1236 may be connected to SB/ICH 1204.

An operating system runs on processing unit 1206. The operating systemcoordinates and provides control of various components within the dataprocessing system 1200 in FIG. 12. As a client, the operating system maybe a commercially available operating system such as Microsoft® Windows10®. An object-oriented programming system, such as the Java™programming system, may run in conjunction with the operating system andprovides calls to the operating system from Java™ programs orapplications executing on data processing system 1200.

As a server, data processing system 1200 may be, for example, an IBMeServer™ System p® computer system, Power™ processor based computersystem, or the like, running the Advanced Interactive Executive (AIX®)operating system or the LINUX® operating system. Data processing system1200 may be a symmetric multiprocessor (SMP) system including aplurality of processors in processing unit 1206. Alternatively, a singleprocessor system may be employed.

Instructions for the operating system, the object-oriented programmingsystem, and applications or programs are located on storage devices,such as HDD 1226, and may be loaded into main memory 1208 for executionby processing unit 1206. The processes for illustrative embodiments ofthe present invention may be performed by processing unit 1206 usingcomputer usable program code, which may be located in a memory such as,for example, main memory 1208, ROM 1224, or in one or more peripheraldevices 1226 and 1230, for example.

A bus system, such as bus 1238 or bus 1240 as shown in FIG. 12, may becomprised of one or more buses. Of course, the bus system may beimplemented using any type of communication fabric or architecture thatprovides for a transfer of data between different components or devicesattached to the fabric or architecture. A communication unit, such asmodem 1222 or network adapter 1212 of FIG. 12, may include one or moredevices used to transmit and receive data. A memory may be, for example,main memory 1208, ROM 1224, or a cache such as found in NB/MCH 1202 inFIG. 12.

As mentioned above, in some illustrative embodiments the mechanisms ofthe illustrative embodiments may be implemented as application specifichardware, firmware, or the like, application software stored in astorage device, such as HDD 1226 and loaded into memory, such as mainmemory 1208, for executed by one or more hardware processors, such asprocessing unit 1206, or the like. As such, the computing device shownin FIG. 12 becomes specifically configured to implement the mechanismsof one or more of the illustrative embodiments and specificallyconfigured to perform the operations and generate the outputs describedherein with regard one or more of the core finding lexicon development,fine-grained finding descriptor generation, ML/DL computer modeltraining, and automated medical imaging report generation, in accordancewith one or more of the illustrative embodiments.

Those of ordinary skill in the art will appreciate that the hardware inFIGS. 11 and 12 may vary depending on the implementation. Other internalhardware or peripheral devices, such as flash memory, equivalentnon-volatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIGS. 11 and 12.Also, the processes of the illustrative embodiments may be applied to amultiprocessor data processing system, other than the SMP systemmentioned previously, without departing from the spirit and scope of thepresent invention.

Moreover, the data processing system 1200 may take the form of any of anumber of different data processing systems including client computingdevices, server computing devices, a tablet computer, laptop computer,telephone or other communication device, a personal digital assistant(PDA), or the like. In some illustrative examples, data processingsystem 1200 may be a portable computing device that is configured withflash memory to provide non-volatile memory for storing operating systemfiles and/or user-generated data, for example. Essentially, dataprocessing system 1200 may be any known or later developed dataprocessing system without architectural limitation.

FIG. 13 is a flowchart outlining an example operation for generatingfine-grained finding descriptor data structures from medical imagingreports and using those fine-grained finding descriptor data structuresto train a machine learning computer model in accordance with oneillustrative embodiment. The operation outlined in FIG. 13 may beperformed, for example, by one or more specifically configured computingdevices of one or more data processing systems, which are specificallyconfigured to implement the core finding lexicon development computingtool 1120, the fine-grained finding descriptor generation computing tool1130, and the machine learning/deep learning (ML/DL) computer modeltraining computing tool 1140 in FIG. 11 and their correspondingcomputing operations to develop a core finding lexicon, use thedeveloped core finding lexicon to generate fine-grained findingdescriptors that define fine-grained finding labels (FFLs), and train aML/DL computer model to perform an AI operation based on the FFLs andtheir fine-grained finding descriptor data structures.

As shown in FIG. 13, the operation starts by performing natural languageprocessing and computer textual analysis on a first corpus of medicalimaging report data structures to extract core findings and coremodifiers used in natural language content or text of medical imagingreports (step 1310). The extracted core findings and core modifiers areevaluated through an automated and/or semi-automated process to identifya subset of core findings and core modifiers to be retrained as part ofa core finding lexicon or vocabulary (step 1320). The core findinglexicon/vocabulary may include the core finding and coremodifiers/modifier types, as well as other information associated withthe core findings, such as finding type or the like.

The core finding lexicon/vocabulary is pre-processed using a smallestprefix building algorithm (step 1330) and the prefix strings are used asa basis to search, using a dynamic programming algorithm, such as alongest common subfix (LCF) based algorithm, for instances of the prefixstrings in text of relevant sections of medical imaging reports, e.g.,the indications and findings sections of medical imaging reports, togenerate vocabulary phrases (step 1340). The vocabulary phrases are usedas a basis for performing a vocabulary-driven phrasal detectionoperation that identifies core finding phrases and helper phrases andassociates core findings with modifiers based on these detected phrases(step 1350). The modifiers in the illustrative embodiments describedherein may be any clinical attribute that is descriptive of the corefinding and thus, indicates a fine-grained specific type of the corefinding. For example, the modifiers may specify clinical attributes suchas laterality, anatomical location, severity, appearancecharacteristics, and the like.

Extended negation detection, extended by the use of pre- andpost-negation term identification operations, is performed on thenatural language content or text corresponding the instances of corefindings and modifiers found in medical imaging reports through theabove operations (step 1360). Based on the results of the association ofcore findings with modifiers, the core finding lexicon, and the extendednegation detection, fine-grained finding descriptor data structures aregenerated for defining fine-grain descriptors or labels (FFLs) (step1370). All non-duplicative descriptors, or a subset of the generatedfine-grained finding descriptor data structures as determined inaccordance with predefined selection criteria, may be maintained in adatabase for training machine learning/deep learning (ML/DL) computermodels (step 1380). Thereafter, the database is used, along with machinelearning training logic, to train one or more ML/DL computer modelswhich are then deployed to perform artificial intelligence (cognitivecomputing) operations, such as medical image analysis, medical imageaugmentation (or labeling), automated patient summary generation basedon patient electronic medical records, or automated medical imagingreport generation (described hereafter) (step 1390). The operation thenterminates.

Thus, the illustrative embodiments provide mechanisms for computerexecuted automatic learning of fine-grained finding labels (FFLs) frommedical imaging report data structures and automatic generation ofdescriptor data structures that can be used to train machinelearning/deep learning models to identify instances of such FFLs orpatterns representative of such FFLs in other textual and/or image inputdata. This automated improved computing tool provides an improvedcomputing tool methodology that permits a relatively small set ofcoarse-grained core findings to be used to automatically learn a largerset of fine-grained findings. The fine-grained findings then permitmachine learning/deep learning models to be trained to identify muchmore specific structures/anomalies and provide more detailed informationabout such specific structures/anomalies. As a result, more focused andaccurate information is able to be provided to medical practitioners,which in turn reduces sources of error in treatment of patients.

Automated Medical Imaging Report Generation

The training of machine learning/deep learning models based on FFLs maybe used to perform various artificial intelligence and cognitivecomputing tasks as noted previously. As an additional feature of someillustrative embodiments, the training of machine learning/deep learningmodels may be employed as part of an artificial intelligence/cognitivecomputing system that operates to automatically generate medical imagingreports based on an input medical image. It should be appreciated thatfor this illustrative embodiment, the FFLs need not be generated usingthe previously described mechanisms and may be provided through othermeans. For example, the FFLs may be manually populated in someillustrative embodiments rather than having an automated mechanism aspreviously described which generates the FFLs based on the core findinglexicon. Thus, while illustrative embodiments for automated medicalimaging report generation will be described where the FFLs are generatedusing the automated mechanisms previously described, the presentinvention is not limited to such and there are other embodimentscontemplated which include the inventive features described hereafter,but with other sources of FFLs utilized. The automated medical imagingreport generation does not require the automated mechanisms forgeneration of an FFL pattern database as previously described above.

Automated medical imaging report generation can greatly assist medicalpractitioners by providing improved computing tools that can quickly andaccurately identify findings in medical images that should be brought tothe attention of the medical practitioner and/or patient so thatappropriate treatments may be evaluated to improve the medical conditionof the patient. With advancements in artificial intelligence (AI), suchas the machine learning/deep learning computer models and mechanismssuch as those described herein, computing tools may be developed toperform automated preliminary reads of medical imaging data which canexpedite clinical workflows, improve accuracy, and reduce overall costs.However, known mechanism for image captioning in computer vision arelimited to a predefined set of semantic topics or limited coarse grainedfindings. Such mechanisms are not clinically acceptable as they do notensure the correct detection of a comprehensive set of findings nor thedescript of their clinical attributes, such as laterality, anatomicallocation, severity, etc. To the contrary, the focus of known mechanismsis on the report language generation rather than the visual detection offindings.

In further illustrative embodiments of the present invention, mechanismsare provided for performing automated medical imaging report generationbased on fine-grained finding labels learned through an automatedlearning process, such as that described previously. As mentioned above,in one illustrative embodiment, the mechanisms described above forgenerating the database of FFL descriptor data structures may be used totrain ML/DL computer models, such as neural networks or the like, forperforming fine-grained label detection in medical image data inputwhich determines, for a given input medical image, which fine-grainedlabels (FFLs) are indicated by image features extracted from the inputmedical image. That is, feature extraction is performed on the medicalimage in a manner generally known in the art, and these features arethen input to a trained ML/DL computer model that associates the patternof features with a classification corresponding to a FFL descriptor datastructure generated through a process corresponding to one or more ofthe illustrative embodiments described above. Again, an example of aML/DL computer model trained for performing such operations is describedin FIG. 10 above. As discussed previously, in some illustrativeembodiments, two models are trained in this manner, one based on corefinding labels and another on the FFL labels that have support in apredetermined number of medical images, e.g., 100, such that thetraining exploits the mutually reinforcing nature of the coarse-finelabels.

With regard to automated medical imaging report generation, a medicalimaging report can be described, in terms of the FFL detection mechanismpreviously described, as a binary pattern vector P={I_(P)(F_(j))} whereI_(P)(F_(j))=1 if the FFL label F_(j)ϵF is present in the report andI_(P)(F_(j))=0 otherwise, this is also referred to herein as a FFLpattern vector P. Here F is the set of FFL labels used in training theML/DL computer model(s) and the binary pattern vector P may have avector slot for each FFL, whose value is set to either 1 or 0 dependingon whether or not the FFL is predicted to be applicable to the extractedimage features from the medical image.

During the medical imaging report database creation process, medicalimaging report data structures characterized by the same binary FFLpattern vector P are collected and are ranked based on the supportprovided by their constituent portions of natural language content,e.g., sentences. Let R_(p)=r_(s) be the collection of reports spanned bya FFL pattern vector P (i.e. the collection of reports having the sameFFL pattern vector P), where again the FFL pattern vectorP={I_(P)(F_(j))}, then Rank(r_(s))=Σ_(j=1) ^(M) ^(s) (s_(j)) where M_(s)is the number of relevant constituent portions of natural languagecontent (e.g., sentences) in report r_(s) spanned by one or more of theFFLs in the pattern P. Here h(s_(j)) is given by

${{h\left( s_{j} \right)} = \frac{{{Number}\mspace{14mu}{of}\mspace{14mu}{reports}\mspace{14mu} r_{i}},{{that}\mspace{14mu}{contain}\mspace{14mu} s_{j}}}{R_{P}}},$

where sj is the portion of natural language content, e.g., sentence,that contains one or more of the FFL patterns. The highest rankedmedical imaging reports are then stored as associated reports with thebinary pattern vectors in a database, e.g., the top ranked medicalimaging report, or top N ranked medical imaging reports for each FFLpattern vector, are stored in association with the FFL pattern vector.

An overall automated medical imaging report generation workflow isillustrated in FIG. 14A. As shown in FIG. 14A, a medical image datastructure 1410 is fed to the two ML/DL computer models, e.g., trainedneural networks, 1420 and 1430. A first one of the ML/DL computer models1420 is trained for classifying image features extracted from themedical image data structure 1410 with regard to core finding labels(CFLs), whereas a second one of the ML/DL computer models 1430 istrained to classify features extracted from the medical image datastructure 1410 with regard to FFLs, such as the FFLs defined by thefine-grained finding descriptor data structures of a database generatedin a manner according to one or more of the illustrative embodimentspreviously described above.

The classification predictions generated by the ML/DL computer models1430 are input to the fusion computer model 1440 where they are“thresholded” using the image based precision-recall F1-score foroptimization. Thresholding is used to convert the real-number predictionscores of the ML/DL computer models 1430 to the binary scores ofpositives and negatives. Let 0 be a vector that contains all labelthresholds. To compute the optimal thresholds, an objective functionbased on the image-based F1 score is used:

${L(\theta)} = {- {\ln\left( {\frac{1}{n}{\sum\limits_{i = 1}^{n}\;{F\; 1_{i}(\theta)}}} \right)}}$

with F1_(i) being the F1 score of image i and n being the number ofimages. The F1 score is the harmonic mean of the positive predictivevalue (PPV) and sensitivity, which is computed as:

${F\; 1} = \frac{{2{TP}} + \epsilon}{{2{TP}} + {FP} + {FN} + \epsilon}$

Where TP, FP, and FN are the true positives, false positives, and falsenegatives, respectively, computed between the ground truth and thebinary scores after thresholding by θ. The value ϵ=10⁻⁷ is used tohandle the 0/0 situation when there are no positives in both predictionand ground truth. The optimal θ can be computed by minimizing L(θ)through an optimization algorithm. In one illustrative embodiment, thederivative-free global optimization algorithm, ESCH, is used as itprovides the best results in tested algorithms. By focusing on thepositive occurrences of findings per image and minimizing L(θ) it isensured that the prediction has as few false positives while stillenabling the detection of relevant findings.

The resulting pattern vectors are combined to result in the consolidatedFFL pattern vector Q={I_(Q)(F_(j))} such that each CFL/FFL in theoutputs of the ML/DL computer models is represented in correspondingvector slots of the consolidated FFL pattern vector Q. The best matchingmedical imaging reports from a medical imaging report database 1460 arethen derived by the FFL pattern and report retrieval engine 1470 fromthe semantically nearest FFL pattern vectors in the FFL pattern database1450. It should be noted that the FFL pattern database 1450 may be thedatabase of fine-grained finding descriptor data structures generatedthrough one or more of the illustrative embodiments described above.

The semantic distance between a query FFL bit pattern vector Q,generated by the fusion module 1440 and a pattern vector P from the FFLpattern database 1450 is given by:

${d\left( {Q,P} \right)} = \frac{\sqrt{\sum\limits_{i = 1}^{F}\;{\omega_{l}\left( {{I_{P}\left( F_{l} \right)} - {I_{Q}\left( F_{l} \right)}} \right)}^{2}}}{F}$

where ω_(l) is the weight associated with the FFL label F_(l). Acriticality rank for each core finding on a scale of 1 to 10 may besupplied by a SME, which may then be normalized and used to weight theclinical importance of a finding during matching. Once the matching FFLpattern in the database 1450 is determined, the FFL pattern and reportretrieval engine 1470 determines the highest ranked medical imagingreport in the report database 1460 based on a ranking of the medicalimaging reports associated with the matching FFL pattern. The ranking ofthis subset of medical imaging reports from the report database 1460 maybe performed in accordance with the ranking function Rank(r_(s))=Σ_(j=1)^(M) ^(s) h(s_(j)), discussed previously, for example.

Having identified the best matching, or highest ranking, medical imagingreport, the FFL pattern and report retrieval engine 1470 drops allsentences from the retrieved report whose evidence cannot be found inthe FFL label pattern of the query Q, thus achieving the variety neededin returned reports per query.

FIG. 14B provides an example diagram of the overall automated medicalimaging report generation workflow of FIG. 14A with some additionaldetails regarding the report database 1460 preparation depicted.Referring to FIG. 14B, each of the prior report electronic documents1480, represented as report clusters R1, R2, . . . RM which are clustersor collections of extracted portions of natural language content, e.g.,sentences. Some portions/sentences can contain more than one FFL patternsuch as s1 in R1 containing both F1 and F2, shown by the arrows in thereports per FFL vector 1484 generated as a result of the extraction offine-grained labels and sentences from the prior reports 1480. Eachcluster R1, R2, . . . RM in the reports per FFL vector 1484 spans asingle FFL vector {F1 . . . FN}. The sentences collected across all thereports in the cluster are ranked based on their frequency of occurrencewithin the cluster (see 1486 in FIG. 14). Thus, s1 is ranked highestbecause it occurs the most across reports R1, R2, . . . RM (eachoccurrence per report is counted once in this analysis so that sentencerepeats within a single report are avoided). The scores of each sentenceare then added for all the sentences in a report R1, R2, . . . RM toobtain an overall score for the report. Note that all reports in thecluster have the same set of FFL patterns occurring (that is thedefinition of the cluster). Thus, the only point of selection ofsentence is now down to those that are most likely to be included inprior reports 1480. In addition, since the ranked sentences are chosenfrom the highest ranked report R1, R2 . . . RM, after selecting thehighest ranked report (R1 in this case), they are also sentences thatare likely to come from a coherent report. Since not all sentences ofthe report R1 are selected, rather only those that cover the FFL patternvector presence, extraneous sentences from the original report areavoided. Thus the quality of the automatically generated report 1488 ishigh.

The workflows shown in FIGS. 14A-14B may be implemented by one or morespecifically configured computing devices of one or more data processingsystems, such as those previously described above. Again, the specificconfiguration of these computing devices, along with the non-genericcomputer operations required to perform the operations described herein,renders these computing devices and data processing system specializedcomputing devices and data processing systems that are specificallyconfigured to implement the mechanisms and perform the operations of theillustrative embodiments which provide an improved computing tool andimproved computing tool operations. These operations are complexcomputer operations specifically involving artificial intelligence(cognitive computing) logic, pattern analysis, and the like, all ofwhich are computer specific non-generic computer operations that cannotbe practically performed in the human mind.

For example, in a distributed data processing environment such as theexample environment shown in FIG. 11, one or more server computingdevices 1104 may be configured to perform the automated medical imagingreport generation workflow shown in FIG. 14. In such an implementation,the medical image data input 1410 may be obtained from any suitablesource computing system, e.g., network attached data storage 1106, oneor more client computing devices 1110-1112, another server computingdevice 1104, or the like. For example, a client computing device 1110may be a computing device at a medical imaging equipment location whichperforms the examination of the patient to capture the medical imagedata and provides the medical image data to the specially configuredserver computing device 1104 via one or more data networks, forautomated medical imaging report generation. The specially configuredcomputing device 1104 may then perform the workflow of FIG. 14 toautomatically generate a medical imaging report data structure which maybe returned to the client computing device 1110 and/or distributed toother authorized computing devices, such as a patient's primary carephysician's office computing devices. Any known or later developedmanner of electronic communication may be used, for example, to exchangethe data between the computing systems and/or distribute theautomatically generated medical imaging report. Of course, appropriateprivacy protection mechanisms, such as encryption and the like, can beused to ensure the privacy of the patient's personally identifiableinformation in any data exchanged, such as in the automaticallygenerated medical imaging report.

By specifically configuring one or more computing devices of one or moredata processing systems to perform automated medical imaging reportgeneration, an improved computer tool and improved computer tool processis provided that provides significant benefits to medical practitioners.Specifically, being able to obtain automated preliminary read reportsfor common examinations, such as chest X-rays, MRIs, CT scans, and thelike, will expedite clinical workflows, improve accuracy in suchclinical workflows minimizing human error, and improve operationalefficiencies of hospitals and medical practices.

The quality of the automatically generated medical imaging reportsgenerated by the mechanisms of the illustrative embodiments isillustrated in FIG. 15 which shows examples of medical images and thecorresponding medical imaging reports generated by manual processes 1510and by the automated mechanisms 1520 of the illustrative embodiments. Ascan be seen from the examples in FIG. 15, the medical imaging reportgenerated by the automated mechanism of the illustrative embodimentsprovides the pertinent finding information as well as modifiers and thelike for finely identifying the findings in the accompanying medicalimages. While the automated reports generated differ in the text used todescribe the medical concepts, they provide an equivalent level ofdetail and information about the same critical medical concepts that areidentified in the manually generated reports, but instead usingdifferent automated tools and automated operations that do not requireintervention of a human subject matter expert.

FIG. 16 is a flowchart outlining an example operation for automatedmedical imaging report generation in accordance with one illustrativeembodiment. The operation outlined in FIG. 16 may be performed by one ormore computing devices of one or more data processing systems that arespecifically configured to perform the workflow outlined in FIG. 14above.

As shown in FIG. 16, the operation starts by training a first and secondML/DL computer model based on core finding labels (CFLs) andfine-grained finding labels (FFLs) to thereby generate a pair of trainedML/DL computer models (step 1610). In one illustrative embodiments, theML/DL computer models may be instances of the ML/DL computer modeldescribed previously with regard to FIG. 10, where one instance istrained using CFLs and another instance is trained using FFLs. The FFLsused as a basis for the training of one of these instances may be FFLsdefined in fine-grained finding descriptor data structures generatedthrough a process as previously described above. Alternatively, in someillustrative embodiments, the FFLs may be curated in a manner similar tothe way that core findings are curated in an automated or semi-automatedmanner for generating the core findings lexicon/vocabulary, as describedabove. In such an embodiment, the core findings represent findingswithout modifiers or with a small limited set of core modifiers, whereasthe FFLs have a more comprehensive listing of modifiers and furtherindicate negative/positive indicators of the core finding.

An input medical image is received from a requestor computing devicethat is requesting that an automated preliminary read of the medicalimage be performed and a corresponding automatically generated medicalimaging report be provided (step 1620). The received input medical imageis input to both of the trained ML/DL computer models for processing(step 1630) with each ML/DL computer model outputting an output vectorindicating a classification of the medical image with regard to apredetermine set of classes corresponding to core findings for the firstML/DL computer model, and fine grained findings for the second ML/DLcomputer model (step 1640). Thus, the CFL trained ML/DL computer model(first ML/DL computer model) outputs a CFL bit vector and the FFLtrained ML/DL computer model (second ML/DL computer model) outputs a FFLbit vector.

The output vectors are combined by a fusion module to generate a revisedFFL output vector (step 1650) which is provided as input to an FFLpattern and report retrieval engine. The FFL pattern and reportretrieval engine searches an FFL pattern database, such as thefine-grained finding descriptor database, to find a matching FFL pattern(step 1660) and identify the associated medical imaging reports from areports database (step 1670). The FFL pattern and report retrievalengine identifies a highest ranking medical imaging report associatedwith the matching FFL pattern as a raw medical imaging report for theinput medical image data (step 1680). The raw medical imaging report isthen processed to remove sentences in the medical imaging report forwhich there is no evidence in the revised FFL vector (step 1690). Theresulting modified medical imaging report is then returned as theautomatically generated medical imaging report data structure for theinput medical image data (step 1700). The operation then terminates.

Thus, in these further illustrative embodiments, mechanisms are providedfor automatically performing preliminary reads of medical images andautomatically generating corresponding medical image reports for use bymedical practitioners. These mechanisms greatly improve automatedcomputer based medical image analysis and automated computer basedmedical image report generation as well as medical practice by providingmechanisms to expedite clinical workflows, improve accuracy of clinicalworkflows, and improve operational efficiencies.

As noted above, it should be appreciated that the illustrativeembodiments may take the form of an entirely hardware embodiment, anentirely software embodiment or an embodiment containing both hardwareand software elements. In one example embodiment, the mechanisms of theillustrative embodiments are implemented in software or program code,which includes but is not limited to firmware, resident software,microcode, etc.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a communication bus, such as a system bus,for example. The memory elements can include local memory employedduring actual execution of the program code, bulk storage, and cachememories which provide temporary storage of at least some program codein order to reduce the number of times code must be retrieved from bulkstorage during execution. The memory may be of various types including,but not limited to, ROM, PROM, EPROM, EEPROM, DRAM, SRAM, Flash memory,solid state memory, and the like.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening wired or wireless I/O interfaces and/orcontrollers, or the like. I/O devices may take many different formsother than conventional keyboards, displays, pointing devices, and thelike, such as for example communication devices coupled through wired orwireless connections including, but not limited to, smart phones, tabletcomputers, touch screen devices, voice recognition devices, and thelike. Any known or later developed I/O device is intended to be withinthe scope of the illustrative embodiments.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems orremote printers or storage devices through intervening private or publicnetworks. Modems, cable modems and Ethernet cards are just a few of thecurrently available types of network adapters for wired communications.Wireless communication based network adapters may also be utilizedincluding, but not limited to, 802.11 a/b/g/n wireless communicationadapters, Bluetooth wireless adapters, and the like. Any known or laterdeveloped network adapters are intended to be within the spirit andscope of the present invention.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The embodiment was chosen and described in order to bestexplain the principles of the invention, the practical application, andto enable others of ordinary skill in the art to understand theinvention for various embodiments with various modifications as aresuited to the particular use contemplated. The terminology used hereinwas chosen to best explain the principles of the embodiments, thepractical application or technical improvement over technologies foundin the marketplace, or to enable others of ordinary skill in the art tounderstand the embodiments disclosed herein.

1. A method, in a data processing system specifically configured toimplement a fine-grained finding descriptor generation computing toolthat automatically generates fine-grained labels as a basis fordownstream computer system operations, the method comprising:processing, by the fine-grained finding descriptor generation computingtool, medical report natural language content of at least one medicalimaging report data structure associated with at least one medicalimage, based on a core finding lexicon data structure, to extract a setof core finding instances of one or more core findings in the corefinding lexicon data structure, from the medical report natural languagecontent, wherein the one or more core findings are terms describing oneof anatomical structures or abnormalities present in the at least onemedical image; executing, by the fine-grained finding descriptorgeneration computing tool, for each core finding instance in theextracted set of core finding instances, automated computer naturallanguage processing operations comprising: generating a parse tree datastructure for a corresponding portion of the medical report naturallanguage content corresponding to the core finding instance,automatically executing phrasal grouping computer operations on theparse tree data structure to thereby associate one or more modifiers ofcore findings specified in the portion of the medical report naturallanguage content with the core finding instance, wherein the one or moremodifiers are terms further defining a characteristic of the corefinding; and generating, by the fine-grained finding descriptorgeneration computing tool, a fine-grained finding descriptor datastructure for the core finding instance based on the association of oneor more modifiers of the core finding with the core finding instance;and storing the fine-grained finding descriptor data structure in afine-grained finding descriptor database for downstream computing systemoperations.
 2. The method of claim 1, wherein the fine-grained findingdescriptor data structure comprises data portions specifying a value ofa core finding which corresponds to the core finding instance, a findingtype indicating a type of the core finding modified by the one or moremodifiers, a negativity indicator indicating whether or not the corefinding modified by the one or more modifiers is negatively indicated inthe portion of medical report natural language content, and the one ormore modifiers of the core finding.
 3. The method of claim 2, whereinthe value of the core finding is a core finding value from the corefinding lexicon data structure, and wherein the core finding lexicondata structure is developed through an automated or semi-automatedprocess implementing automated computerized natural language processingcomputer tools to analyze and extract features from natural languagecontent of a corpus of medical imaging report data structures indicatingvalues of core findings.
 4. The method of claim 1, wherein processingthe medical report natural language content comprises: building an indexdata structure having entries for synonyms of core findings in the corefinding lexicon data structure, and wherein the entries for synonyms ofcore findings point to corresponding core findings in the core findinglexicon data structure; building a core findings prefix data structurespecifying prefixes of terms within core findings of the core findinglexicon data structure for matching with text such that a portion oftext containing a prefix associated with a core finding is determined tobe specifying the core finding in the portion of text; and processingthe medical report natural language content based on the index datastructure and core findings prefix data structure to identify instancesof core findings, instances of synonyms of core findings, and instancesof prefixes of core findings in the at least one medical imaging reportdata structure, to thereby generate the set of core finding instances.5. The method of claim 4, wherein processing medical report naturallanguage content based on the index data structure and core findingsprefix data structure comprises executing a longest common subfix (LCF)algorithm on portions of the medical report natural language content toidentify portions of medical report natural language content having textmatching one or more core findings prefixes specified in the corefindings prefix data structure.
 6. The method of claim 1, whereinautomatically executing phrasal grouping computer operations on theparse tree data structure comprises grouping nodes of the parse treedata structure into one of a core phrasal group or a helper phrasalgroup, wherein nodes of a sub-tree that comprise a node corresponding toa core finding from the core finding lexicon data structure are groupedinto a core phrasal group, and nodes that are not part of a sub-treecomprising a node corresponding to a core finding are grouped into oneor more helper phrasal groups.
 7. The method of claim 6, whereinexecuting the phrasal grouping computer operation comprises merging twoadjacent core phrasal groups, in the parse tree data structure, when thetwo adjacent core phrasal groups are determined to contain a nodecorresponding to a same core finding from the core finding lexicon datastructure.
 8. The method of claim 6, wherein executing the phrasalgrouping operation comprises for each helper phrasal group, associatingany modifier present in the helper phrasal group with core findings inone or more core phrasal groups that are adjacent to the helper phrasalgroup in the parse tree data structure.
 9. The method of claim of claim1, wherein the automated computer natural language processing operationsfurther comprises performing negated instance detection of core findingsat least by performing a search of the parse tree data structure basedon a set of known negation keywords and negation term patterns, and foreach instance of a negation keyword or negation term pattern,determining a scope of nodes in the parse tree data structureencompassed by the instance of the negation keyword or negation termpattern, and wherein core findings located within a scope of nodes inthe parse tree data structure encompassed by the instance of thenegation keyword or negation term pattern are determined to benegatively indicated.
 10. The method of claim 9, wherein performingnegated instance detection further comprises retrieving a set ofnegation prior terms and negation post terms, and searching the parsetree data structure for instances of nodes having terms matching termsin one or more of the set of negation prior terms or negation postterms, and wherein a core finding located in a node corresponding to aportion of natural language text corresponding to an instance of a termmatching a term in one or more of the set of negation prior terms ornegation post terms, is determined to be negatively indicated.
 11. Themethod of claim 10, wherein generating the fine-grained findingdescriptor data structure comprises setting a negation indicator in thefine-grained finding descriptor data structure to indicate thefine-grained finding to be negatively indicated in response to thenegated instance detection indicating that the core finding isnegatively indicated.
 12. The method of claim 1, wherein the downstreamcomputer system operations comprise training a machine learning computermodel executed on one or more computing devices, by performing machinelearning on the machine learning computer model using the fine-grainedfinding descriptor database to provide fine-grained finding labels forfindings in medical imaging reports.
 13. The method of claim 12, whereinthe machine learning computer model is trained to perform automatedfine-grained labeling of medical image data structures representingmedical images, based on fine-grained finding labels extracted fromassociated medical imaging reports and the fine-grained findingdescriptor data structures of the fine-grained finding descriptordatabase.
 14. The method of claim 1, wherein the core finding lexicondata structure specifies core findings and core finding types, andwherein generating the fine-grained finding descriptor data structurefor the core finding instance based on the association of one or moremodifiers of the core finding with the core finding instance furthercomprises combining the one or more modifiers and core finding instancewith the core finding type corresponding to the core finding instancefrom the core finding lexicon data structure, to thereby generate afine-grained finding descriptor data structure.
 15. The method of claim1, further comprising: determining, for each given instance in a set ofdifferent instances of a fine-grained finding descriptor data structure,a count of a number of other instances of fine-grained findingdescriptor data structures that match the given instance; and comparingthe counts for each given instance to a threshold value to select asubset of given instances of fine-grained finding descriptor datastructures for inclusion in the fine-grained finding descriptordatabase, wherein only given instances of fine-grained findingdescriptor data structures whose counts equal or exceed the thresholdvalue are stored in the fine-grained finding descriptor database. 16.The method of claim 1, wherein the at least one medical image is atleast one of a human chest radiology image, and wherein the at least onemedical imaging report data structure is a medical imaging reportspecifying indications, findings, and impression of the at least onemedical image generated by a subject matter expert.
 17. The method ofclaim 1, wherein executing, by the fine-grained finding descriptorgeneration computing tool, for each core finding instance in theextracted set of core finding instances, automated computer naturallanguage processing operations further comprises identifying a subset ofrelevant sections of the natural language content of the at least onemedical imaging report data structure where core findings are likely tobe found, and wherein the automated computer natural language processingoperations are performed on the portions of the medical report naturallanguage content associated with the subset of relevant sections. 18.The method of claim 17, wherein the relevant sections are an indicationssection and a findings section of the at least one medical imagingreport data structure.
 19. A computer program product comprising acomputer readable storage medium having a computer readable programstored therein, wherein the computer readable program, when executed ona computing device, causes the computing device to implement afine-grained finding descriptor generation computing tool thatautomatically generates fine-grained labels for downstream computersystem operations at least by: processing medical report naturallanguage content of at least one medical imaging report data structureassociated with at least one medical image, based on a core findinglexicon data structure, to extract a set of core finding instances ofone or more core findings in the core finding lexicon data structure,from the medical report natural language content, wherein the one ormore core findings are terms describing one of anatomical structures orabnormalities present in the at least one medical image; executing, foreach core finding instance in the extracted set of core findinginstances, automated computer natural language processing operationscomprising: generating a parse tree data structure for a correspondingportion of the medical report natural language content corresponding tothe core finding instance; automatically executing phrasal groupingcomputer operations on the parse tree data structure to therebyassociate one or more modifiers of core findings specified in theportion of the medical report natural language content with the corefinding instance, wherein the one or more modifiers are terms furtherdefining a characteristic of the core finding; and generating, by thefine-grained finding descriptor generation computing tool, afine-grained finding descriptor data structure for the core findinginstance based on the association of one or more modifiers of the corefinding with the core finding instance; and storing the fine-grainedfinding descriptor data structure in a fine-grained finding descriptordatabase for use in downstream computer system operations.
 20. Anapparatus comprising: a processor; and a memory coupled to theprocessor, wherein the memory comprises instructions which, whenexecuted by the processor, cause the processor to implement afine-grained finding descriptor generation computing tool thatautomatically generates fine-grained labels for downstream computersystem operations at least by: processing medical report naturallanguage content of at least one medical imaging report data structureassociated with at least one medical image, based on a core findinglexicon data structure, to extract a set of core finding instances ofone or more core findings in the core finding lexicon data structure,from the medical report natural language content, wherein the one ormore core findings are terms describing one of anatomical structures orabnormalities present in the at least one medical image; executing, foreach core finding instance in the extracted set of core findinginstances, automated computer natural language processing operationscomprising: generating a parse tree data structure for a correspondingportion of the medical report natural language content corresponding tothe core finding instance; automatically executing phrasal groupingcomputer operations on the parse tree data structure to therebyassociate one or more modifiers of core findings specified in theportion of the medical report natural language content with the corefinding instance, wherein the one or more modifiers are terms furtherdefining a characteristic of the core finding; and generating, by thefine-grained finding descriptor generation computing tool, afine-grained finding descriptor data structure for the core findinginstance based on the association of one or more modifiers of the corefinding with the core finding instance; and storing the fine-grainedfinding descriptor data structure in a fine-grained finding descriptordatabase for use in downstream computer system operations.